DOCUMENT RESUME 



ED 274 520 



SE 047 214 



AUTHOR 
TITLE 



INSTITUTION 

SPONS AGENC7 

PUB DATE 
NOTE 

AVAILABLE FROM 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



St owe, Robert C. 

Annotated Bibliography o£ Publications Dealing with 
Qualitative and Quantitative Indicators o£ the 
Quality o£ Science. 

Harvard Univ., Cambridge, MA. Jefferson Physical 
Lab. 

Josiah Macy, Jr. Foundation, New York, N.Y.; National 
Science Foundation, Washington, D.C. 
Jun 86 

lOlp.; A technical memorandum of the Quality 
Indicators Project (Principal Investigator: Gerald 
Holton). 

Quality Indicators Project, Professor Gerald Holton, 
Jefferson Physical Lab. 358, Harvard University, 
Cambridge, MA 02138 ($7.00). 
Reference Materials - Bibliographies (131) 

MF01/PC05 Plus Postage. 

Annotated Bibliographies; ^Evaluation Criteria; 
Females; Peer Evaluation; Researchers; ^Research 
Methodology; ^Research Problems; Research 
Utilization; Science Education; Science History; 
^Scientific Research; Trend Analysis 
^Research Priorities; ^Science Indicators 



ABSTRACT 

Designed to assist in the study of techniques for and 
approaches to the evaluation of scientific research, this annotated 
bibliography of publications deals with qualitative and quantitative 
indicators of the quality of science. Entries are provided in the 
following categories: (1) bibliometric indicators of the quality of 
scientific research; (2) qualitative approaches to and more general 
works on research evaluation; (3) works dealing with science 
indicators; (4) forecasting and research priorities; (5) peer review; 

(6) quality and quantity in the history of science and philosophy; 

(7) education; (8) issues involving quantity and quality in 
particular disciplines (including social indicators, history, health, 
and policy evaluation); (9) sociology of science; (10) methodological 
papers and bibliographies; and (11) access of women to participation 
in scientific research. (ML) 
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(Principal Investigator: Gerald Holton) 



PREFACE 

The following bibliography has been compiled for use by the 
Quality Indicators Project. However, in spite of its undoubted 
incompleteness in many respects, ve have been persuaded to make 
it available to other scholars studyi'hg techniques for and 
approaches to the evaluation of scientific research. On a quid 
pro quo basis, we ask such readers to alert us to literature that 
should be included and to errors in the text. 

. Interest in this field has grown during the last decade for 
several reasons. National budgets for research have shrunk, or 
at least grown less quickly than they had previously, prompting 
inquiry into appropriate means to make choices between projects 
and programs requiring funding. Scientific research leads, by 
winding paths, to technological development and therefore to 
economic benefits; in times of economic turmoil demands are made 
to define the paths and therefore, it is hoped, maximize the 
benefits. The perception has become more widespread that 
scientific research, both in itself and through its technological 
fruits, can have harmful as well as beneficial effects for 
society. Concomitantly, the need has arisen for means to assess 
a priori the impact of research. 

Also, it has become clear that women and minorities are 
underrepresented in the scientific community. Studies of this 
phenomenon, many of them performed by sociologists, often involve 
the determination of the quality of research and researchers. It 
is important to understand as fully as possible the strengths and 
weaknesses of the various approaches to such determinations in 
order that the desired ends— fair science and good science — be 
simultaneously achieved. (This project has in particular 
reviewed the literature dealing with women's participation, or 
lack of participation, in science. Much of this literature is 
listed in Section XI of the bibliography.) 

The Quality Indicators Project is happy to acknowledge 
support and funding from the National Science Foundation and the 



* Jefferson Physical Laboratory, Harvard University, Cambridge, 
Mass. 02138 
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Josiah Macy, Jr. Foxindation. The distribution of this jBemorandum 
does not imply endorsement by either of these foundations. 



TNTRODUCTION 

A. Other Bib liographies 

In the four bibliographies listed below one nay also find 
references to pxablications vhich address various aspects of 
science indicators or the evaluation of research. The four 
bibliographies are held in Project files. 

We have attempted to list in these pages the more important 
citations from these bibliographies, but inevitably some will 
have been neglected. At many points we quote directly from 
another annotated bibliography, except for reference 3 
immediately below. Each quote is accompanied by a reference 
to the original bibliography. The Abt Associates bibliography 
(reference 2 below) contains abstracts more detailed than have 
generally been attempted here and will therefore be valuable 
regardless of any overlap with our compilation. 



1. "Citation analysis: An annotated bibliography," Susan Cozzens, 
et al.. Institute for Scientific information (May 1978). 

Includes 678 entries, explanatory introduction, and an index 
which interfiles authors and subjects. When the notes in 
the current bibliography quote from "Citation analysis" they 
cite the original as "Cozzens." 

2. "Methods for the strategic evaluation of research programs: 
The state of the art — Annotated bibliography," Jonathan D. 
Hodgdon, Stephen J. Fitzsimmons, et al.. Center for Science and 
Technology Policy Studies, Abt Associates, Inc. (March 15, 1985). 

Bibliography containing 47 entries with detailed annotation, 
organized under the following categories: 1. Bibliometric 
methods; 2. Economic and stochastic decision models? 3. Peer 
review and scoring methods; 4. [cross-national] Comparative 
studies; and 5. Other studies. In addition, the 
report includes a reprint of a non-annotated bibliography 
entitled "Documents collected for the federal research 
evaluation activities study" [of the George Washington 
University Program of Policy Studies in Science and 
Technology; see entry under Jjogsdon in Sec. II] . Accompanies 
"Methods for the strategic evaluation of research programs: 
The state-of-the-art (Analytical overview report)" by the 
same authors. 

3. "Productivity of women in science," prepared by the Institute 
for Scientific Information (July 23, 1985). 

Two bibliographies culled from Sociological Abstracts and 
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SCISEARCH, respectively. The former is annotated and 
contains 49 entries; the latter is non-annotated and 
contains 111 entries. Miese compilations are availeible from 
the Institute for Scientific Information. The notes in the 
current bibliography do not quote from '"Productivity of 
women in science . ** 

4. *^Scientometric analyses in physics: A bibliography of 
publication, citation and mobility studies," Jan Vlachy, 
Czechoslovak Journal of Physics , B35 (1985): 1389-1436. 

A very coiEq;>rehensive but non-annotated listing of about 950 
publications. The bibliography is unusual in that somewhat 
more than 30% of the entries are from Eastern European or 
Soviet sources. The compiler, who is one of the world's 
experts in bibliometric assessment of research, offers a 
short but insightful introduction and a list of the major 
sources used, which includes a number of other useful 
bdLbl iographies • 



B. Location of Items 

The name of the Quality Indicators Project file in which our 
copy of a paper is stored is indicated after the reference. For 
example, a particular citation may be followed by "file 
•Citations'" or "file 'Priorities and Foresight'." References to 
books or longer documents are often followed by the term "Shelf" 
rather than by the name of a file. These, clearly, are 
physically stored on the shelves of the project office. A few 
papers and books are not held by the Project. These are usually 
followed by the name of a Harvard University Library (e.g. 
Wldener, CsJDot) and a call number (usually Library of Congress) . 

If the piece is also listed in one or more of the 
bibliographies referred to immediately above, one of the 
following three symbols may be found after a cite: COZZENS, ABT, 
or WOM. "COZZENS" means that the item is also listed in 
"Citation Analysis: An Annotated Bibliography" (reference 1 
above) ; "ABT" means that the item is also in "Methods for the 
strategic evaluation of research programs: The state of the 
art—Annotated bibliography" (reference 2 above); "WOM" means 
that the item is also in the Sociological Abstracts search 
conducted by the Institute for Scientific Information (reference 
3 above). Vlachy 's bibliography (reference 4 above) has not been 
cross-*referenced witlt the items in this compilation. We suspect, 
however, that almost all of the items in sections i-lV following, 
plus many more, are also listed in Vlachy' s work. 
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C. Index/Organization of Items 

Items are organized according to the following categories. 
These categories overlap/ and a number of items are listed iinder 
more than one. 

Core books .........1 

I. Bibliometric indicators of the quality of 
scientific research... 2 

A. Citations and publications as indicators of 
quality 2 

B. critiques of citation analysis 14 

C. Citation context analysis 18 

II. Qualitative approaches to and more general works 

on research evaluation 20 

III. Works dealing specifically with "science 
indicators" 35 

IV. Forecasting and research priorities 41 

V. Peer review 48 

VI. Quality and quantity in the history of science 

and philosophy. - 51 

VII. Education. • 58 

VIII. Issues involving quantity and quality in 
particular lisciplines, including papers on 

social indicators 61 

A. Social indicators 61 

B. History 63 

C. Health. > 64 

D. Policy evaluation. 65 

E. Miscellaneous 66 

IX. Sociology of science (including scientific 
communication) < 68 

X. Methodological papers and bibliographies 72 

XI. Access of women to participation in scientific 
research » 

6 
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i. CORE BOOKS 

Elkana^ Yehuda^ et al.^ eds.^ Toward a Metric of Science (N.Y.: 
John Wiley & Sons, 1978) . Shelf. 

Includes papers by Thackray, Holton, Kochen, Griliches, 
Zinan (see Sec. II); Duncan, Price, Kruskal, Garfield, et 
al.. Cole, et al., Zeisal, and Ezrahi (see Sec. III). 

La Follette, Marcel Chotkowski, ed.. Quality in Science 
(Cambridge, Mass.: MIT Press, 1982). Shelf. 

Includes papers by Brooks, R.S. Morison, and Mazlish (see 
Sec. Ill) ; Bransconb, Previtt, Yankelovich, Weingart and Bok 
(not noted separately) ; and shorter pieces by Senator Hatch 
and Congressmen Brown, Fuqua, and Walgren. With an 
introduction by Gerald Holton. 

National Science Board, Science Indicators (Washington, 
D.C.: National Science Foundation, 1972, 1974, =1976, 1978, 
1980, 1982, 1984) . 

Shils, Edward, ed. , Criteria for Scientific Development! Public 
Policy and National Goals (Cambridge, Mass.: MIT Press, 1968). 
Shelf. 

Collection of papers from Minerva by Weinberg, Maddox, 
Toulmin (see Sec. IV) ; Carter, Williams, Rottenberg, 
Polanyi, Dedijer, Moravcsik, and Salam. 

U.S. Congress, Office of Technology Assessment, Research Funding 
as an Investment: Can We Measure the Returns? (Washington, D.C.: 
U.S. Government Printing Office, 1986). 

Literature review of bibliographic and economic approaches 
to evaluating scientific research. Concludes that 
quantitative methods for assessing the returns on research 
are of limited use, especially for research conducted by 
government agencies, and that qualitative approaches, 
especially peer review, are essential for such assessments. 
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I. Bibliometric indicators of the cnialitv of scientific yesearch 

A. citations and publications as indicators of quality 

Bayer, Alan E. and Folger, John, "Some correlates of a citation 
measure of productivity in science," Socioloerv of Education . 39 
(1966): 381-390. File "Citations." COZZENS. 

Authors' z05Stract: "The science Citation Index provides an 
easy way to <Serive criterion measures of scientific 
accomplielia^gnt. Measures derived from citation counts, the 
principle criterion, have high face validity. These 
criterion measures are found to have a low but positive 
correlation with the quality of scientists' graduate 
education [based on Car^ter report] .and no relation to his 
measured l.Q. score." 

Braun, Tiber and Bujdoso Emo, "The growth of modem analytical 
chemistry as reflected in the statistical evaluation of its 
svibject literature," special issue of CRC Critical Reviews in 
Analytical Cheroistirv . 13 (March 1982): 223-312. File 
"Citations." 

Braun is one of the founders of the journal Scientometrics 
and this is his most ambitious scientometric study. He and 
his colleague present and analyze data on the volume, 
growth, "aging," and distribution (with respect to 
countries, languages, topics, and other criteria) of the 
analytical chemistry literature. They also investigate 
publication and citation patterns in analytical chemistry 
journals and of groups of scientists in the field. 

Broadus, Robert N. , "An investigation of the validity of 
bibliographic citations," Journal of the American Society for 
Information Science . 34 (March 1983): 132-135. File "Citations." 

Author's abstract: "[E.O.] Wilson, in his famous work, 
Sociobiolocry^ The New Synthesis . . . , makes reference to a 
pair of articles by W.D. Hamilton, but misquotes the 
articles' title. No less than 148 later papers make 
reference to both Wilson's book and Hamilton's articles, by 
title. Thus, there is provided an opportunity to test the 
charge, made by some critics, that writers frequently lift 
their bibliographic references from other piiblications 
without consulting the original sources. Although 23% of 
these citing papers made the same error as did Wilson, a 
further perusal of the evidence raises considerable doubt as 
to whether fraudulent use was intended." 

Bruer, John T., "Methodological rigor and citation frequency in 
patient compliance literature," American Journal of Public 
Health . 72 (Oct. 1982): 1119-1123. File "Citations." 

Author's abstract: "An exhaustive bibliography which assesses 
the methodological rigor of the patent compliance 
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literature, and citation data from the Science Citation 
Index (SCI) are combined to determine if methodologically 
rigorous papers are used with greater £reg[uency than 
substandard articles by compliance investigators » There are 
low, but statistically significant, correlations between 
methodological rigor and citation indicators. • .The 
correlation is not strong enough to warrant use of citation 
measures as indicators of rigor on a paper-by-paper basis. 
The data do suggest that citation measures might be 
developed as crude indicators of methodological rigor*. 

Bruer, John T. , "Methodological rigour and review citation 
frequency in patient compliance literature," Journal of 
Documentation . 21 (Sept. 1983): 166-170. File "Citations." 

An extension of the research discussed in the paper cited 
immediately above, examining the relationship between 
methodological rigor and citation frequency in review 
articles o 

Bruer, John T. , "Methodological quality and citation frequency of 
the continuing medical education literature," Journal of 
Documentatio n. ^ (Sept. 1985): 165-172. File "Citations." 

Author's abstract: "Bibliometric analysis of research reports 
in continuing medical education that were evaluated for 
their methodological quality is used to assess the relation 
between methodolgical rigour and citation frequency. There 
is a positive, significant but low correlation between 
rigour and citation, and evidence that rigorous work will be 
cited independent of where the article is pxiblished. There 
is no strong evidence that study design is correlated with 
subsequent citation. . .In certain cases citation measures 
might be used as guides to or indicators of methodologically 
rigorous reports." 

Bujdoso, Erno and Braun, Tiber, "Publication indicators of 
relative research efforts in physics subfields," Journal of the 
American Society for Information Science p 34 (March 1983): 
150-155. File "Citations." 

Uses publication counts to compare research "activity" across 
subfields in physics and across countries. 

Castro, Barry, "The scientific opportunities foregone because of 
more readily available federal support for research in 
experimental than theoretical physics," Journal of Political 
Economy, 26, No. 4 (July/August 1968): 601-614« File 
"Citations . " COZZENS. 

"It is hypothesized that relatively easy federal support for 
experimental work causes physicists to avoid theoretical 
problems that have greater potential scientific 
utility. . .Such foregone scientific opportunities are 
regarded as opportunity fiOets of federal support for physics 
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research, and it is the meaBurement of these costs that is 
this paper's basic svibject matter* The net opportxinity 
costs of federal research cannot be measured in money terms, 
and the physics literature is used to provide an alternative 
source of measurement ... any excessive emphasis chat federal 
science-supporting agencies have placed on experimental 
physics should be reflected in a relatively low average 
frequency of citation to experimental articles and a 
relatively high average frequency of citation to theoretical 
articles." The hypothesis is confirmed. 

Chang, K.H., "Evaluation and survey cf a sxabfield of physics: 
Magnetic resonance and relaxation studies in The Netherlands," 
(Utrecht, The Netherlands: Stichting F.O.M., report no. 
FOM-37175, 1975). Shelf. 

Author's abstract: "A survey of the Dutch activities ... and 
the significant publications in the subfield [are 
presented] .. .it is concluded that the various approaches 
which have been followed to evaluate the impact of the 
particular studies, viz. interviews with experts, scanning 
of textbooks, searching for awards, analyzing the Science 
Citation Index, etc., have yielded non-contradictory 
results... A new procedure to analyze data obtained from the 
Science Citation Index is discussed in the appendix." 
Includes a section in the concluding chapter entitled 
"Evaluation: citations, a measure for quality?" Although 
the report emphasizes the value of citation analysis, it 
explores in detail the problems with the iise of this set of 
techniques. 

Chang, Hans and Dieks, Dennis, "The Dutch output of publications 
in physics," in Le Pair and Volger, eds., Phvsics in The 
Netherlands . Vol. II (Utrecht: Foundation for Fundamental 
Research on Matter, 1982): A3-A18. Shelf. 

Is accompanied by a research note on pviblished productivity 
in twenty countries, describing available citation indexes 
and based on work of D.J. de S. Price. 

Cohen, J.E., "Publication rate as a function of laboratory size 
in a biomedical institution," selentvometrlcs. 2, No. 1 (1980): 
35-52. File "Citations." 

Statistical study of the pviblication output of a research 
group at Rockefeller University. Finds that output of group 
is approximately proportional to nvimber of individuals in 
the group. 

Cole, Jonathan R. , "Patterns of intellectual influence in 
scientific research," Seoiolocrv of Education. 43 (1970): 377-403. 
Not in files. COZZENS. 

Cozzens' abstract: "This paper challenges the 'Ortega 
hypothesis'; that is, the assumption that science progresses 
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only through the cimulative efforts of the whole commxinity. 
Using citation coxxnts to determine 84 university physicists' 
best papers as of 1965, the intellectual input to these 
papers is then traced by looking at the citations contained 
in them. Citation is then used as an indicator of quality, 
and as a dociimentation for intellectual influence and use. 
The study indicates that physicists at all strata 
disproportionately cite the work produced by members of the 
most distinguished departments." 

Cole, Jonathan R. and Cole, Stephen, "The Ortega hypothesis," 
Science , 178 (October 27, 1972): 368-374. File "Citations." 
COZZENS . 

Supports the theory that science operates on the basis of 
uhiversalistic criteria rather than elite control. The 
authors find that the "quality of published research [as 
measured by citation counts] explains more variance than any 
other variable on several types of recognition." 

Cole, Stephen, "The growth of scientific knowledge: Theories of 
deviance as a case study," in L.A. Coser, ed« , The Idea of Social 
Structure: Papers in Honor of Robert K. Merton {N.y«: Harcourt 
Brace Jovanovich, 1975). Paper in File "Sociology of Science"; 
book in Social Relations Library HM24 .134. COZZENS. 

Uses Merton' s theory of deviance (social structure and 
anomie) and subsequent literature on deviance "as a research 
site to explore the social processes through which knowledge 
grows and ideas change." Cole builds on the work of Thomas 
Kuhn, who "has concluded that, ultimately, the process 
through which ideas change and develop can be understood 
only through sociological zmalysis." The author outlines 
"some of the problems raised by Kuhn and then, using the 
field of deviance research as an example," illustrates "how 
we might go about empirically investigating a few of these 
problems." This investigation relies on citation analysis. 

Cole, Stephen, "Professional standing and the reception of 
scientific discoveries," American Journal of Sociolocrvr , 76 (Sept. 
1970). File "Sociology of Science." COZZENS. 

Tests the hypothesis that "if the Matthew Effect were to 
operate, the reception of papers of equal quality should be 
influenced by the location of their authors in the 
stratification system." Measures quality with citations. 
Cole finds that, for the most part, the hypothesis is not 
supported by the data. 

Cole, Stephen and Cole, Jonathan R. , "Scientific output and 
recognition: A study in the operation of the reward system of 
science," American Sociological Review , 32 (June 1967): 378-390. 
Pile "Citations." COZZENS. 

Studies "The relationship between the quantity and quality 

o 11 
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Of scientific output of 120 iiniversity physicists. Although 
these two variables are highly correlated, some physicists 
produce many papers of little significance and others 
produce a few papers of great significance." Quality ia 
measured by citations, using SCI . 

Cole, Stephen, et al., "Measuring the cognitive state of 
scientific disciplines," in Elkana, et al., eds,. Toward a^etric 
of Science (N.Y.: Wiley, 1978): 209-251. Shelf. 

From the authors' introduction: "Problems occur in 
interpreting the indicators presented in SI-72 . . .Two fields 
appearing to be similar on a set of quantitative indicators 
may be qualitatively in quite different stages of 
development and exhibit very different cognitive 
structures ... Clearly we need indicators of the qualitative 
as well as the quantitative aspects of science." However, 
the indicators of qualitative aspects of science used 
in the paper are themselves quantitative (pxablications 
and citations) . 

Garfield, Eugene, "Are the 1979 prizewinners of Nobel class?" 
current Contents (Sept. 22, 1980): 5-13. File "Citations." 

Citation analysis of 1979 Nobel winners. S. Weinberg was 
found to have an unusually high number of citations to his 
work. 

Garfield, Eugene, "The awards of science: Beyond the Nobel Prizes — Part 
2: The winners and their most cited papers," current Contents (Dec. 10, 
1984): 3-17. File "Citations." 

Citation analysis of the work of winners of 56 prestigious 
scientific prizes, with qualitative discussion of important 
papers . 

Garfield, Eugene, citation Indexing — Its Theory and Application 
in Science. Technology, and Humanities ( N.Y.: John Wiley & Sons, 
1979). Cabot Science Library Z697 .S5 .G37. 

Garfield's summary of two decades of work in the field. 
Chapters on the use of citation indexing in patent work, 
literature searches, science management, history and 
sociology of science, and "mapping the structure of science" 
(co-citation analysis) . A final chapter examines the 
difficulties and dangers associated with the use of citation 
analysis for the evaluation of individual scientists. 
Includes a foreward by Robert K. Merton. 

Garfield, Eugene, Essays of an info rmation Scientist. Vols. 1-7 
(Philadelphia: ISl Press). Volumes compile essays from the 
years 1962-73, 1974-76, 1977-78, 1979-80, 1981-82, 1983, 1984. 
Shelf. 

Include a great many essays on citation analysis. With 

12 



ERIC 



7 



Forewords by Joshua Lederberg, Derek J. de Sella Price, 
Harriet Zuckerman, Robert K. Merton, V.V. Nalimov, and 
Gerald Hoi ton. 

Garfield, Eugene, "How to use citation analysis for faculty 
evaluations, and when is it relevant?" Parts I and II, Cuyrent 
Contents (Oct. 31, 1983): 5-13; (Nov. 7, 1983): 5-14. File 
"Citations. " 

Proposes a use of citations which is quite controversial. 
Suggests that citation analysis might be useful in 
identifying colleagues who are competent to judge a 
candidates • work. Goes beyond this, however, to discuss 
quantitative approaches to measuring impact of papers, using 
citations of various kinds. 

Garfield, Eugene, "The 100 most-cited papers ever and how we 
select •Citation Classics"," Current Contents (June 4, 1984): 
3-9. File "Citations." 

Garfield notes that "we use citation frequency as the first 
criterion in selcting 'Citation Classics'," then goes on to 
elaborate^ 

Garfield, Eugene, "Price's citation cycle," current Contents 
(Sept. 29, 1980): 5-7. File "Citations" (with Erice) . 

Personal emd intellectual biography of Derek de Solla Price. 
See also Price, "Citation cycle" (Sec. I. A). 

Garfield, Eugene, et al., "Citation data as science indicators," 
in Elkana, et al., eds.. Toward a Metric of Science (N.Y.: Wiley, 
1978): 179-207. Shelf. 

Includes discussions of co-citation and bibliographic 
coupling, the social and cognitive stmicture of science, and 
models of scientific research. 

Goffman, William, "A pragmatic approach to literature selection," 
in K.S. Warren, ed.. Selectivity In Information Systems : 117-143. 
(See Sec. II. ) . 

Discusses an approach to libraries' selection of journals 
using co-citation maps (of journals, not authors) . 

Gordon, Michael D. , "How authors select journals: A test of the 
reward maximization model of submission behavior," Social S tudies 
of Science , 14 (1984): 27-43. File "Citations." 

Tests the hypothesis that scientists choose journals to which 
to submit their papers barbed on expected rewards. Findc 
that hypothesis is not etipported, but rather scientists 
choose journals on the basis of the audiences the journals 
reach. 

13 
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Griffith, Belver C, et al., "The aging of scientific literature: 
A citation analysis," Journal of Docume ntation. IS (Sept. 1979): 
179-196. File "Citations." 

The authors study various mathematical models of the manner 
in which use of scientific documents and journals declines 
over time. Their practical aim is to assist librarians in 
managing collections. They find that a model developed by 
B.C. Brookes confirms well with their citation data, with 
some caveats, including the observation that "Aging depends 
not merely on the material itself, but [also on] its ueer, 
and a single journal may be aged very differently by 
different user communities." 

Inhaber, H. and Przednowek , K. , "Quality of research and the 
Nobel Prizes," social Studies of Science . 6 (1976): 33-50. File 
"Citations." COZZENS. 

Statistically analyzes citation rates and changes in 
citation rates in an attempt to measure the "halo effect" in 
Nobel winners. Compares latter with scientists elected to 
the NAS. Finds that prc^r.tige and reputation have something 
to do with rise in citations after receiving prize. 
See also piece by Kppers, et al., (Sec. II.) which studies 
the Nobel nominating process in a qualitative manner. 

Knudsen, Dean D. and Vaughan, Ted R. , "Quality in graduate 
education: A re-evaluation of the rankings of sociology 
departments in the Cartter report," The American sociologist 
(Feb. 1969): 12-19. File "Education." 

Criticizes the Cartter report for its "subjective 
evaluations" and attempts to discern the correlation 
between these and more "objective indicators." The 
objective indicator used is publications and the authors 
find that the ranking of the best departments so indicated 
corresponds closely with that of the Cartter report, while 
the ranking of departments of lesser quality does not 
correlate well. See also W.J. Moore (Sec. I.A) , Bayer and 
Folger (Sec. I.A), and Beyer and Snipper (Sec. VII). 

Lawani, Stephen M. and Bayer, Alan E. , "Validity of citation 
criteria for assessing the influence of scientific publications. 
New evidence with peer assessment," Journal of the American 
Society fo-r Informal-Ion Science . 34 (January 1983): 59-66. File 
"Citations." 

Authors' abstract: "This article reviews the principle 
correlational studies employing citation counts as criterion 
measures for assessing the impact of scientific scholarship. 
The rationale and limitations of such measures and studies 
are discussed. New evidence on the validity of citation 
criteria is presented based on a sample of 870 cancer 
research papers. . .Results consistently show that highly 
rated papers are more highly cited over the ensuing five 
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years after publication, or when controls are introduced for 
self -citations, for the influence of listing in the 
yearbook, and for language and country of authorship. The 
implications of results are discussed." 

Lewin, J., "A quantitative and qualitative case study analysis of 
scientific productivity in agricultural research," Israel Journal 
of Agricultural Research , g2 (1972): 129-139» Not in files. 
COZZENS . 

Cozzens' Abstract: "The pre~l971 publication output of a 
group of 37 soil scientists* are analyzed in relation to 
the age of the research group. A steady increase in 
publication output with age is noted... A naxinuia [mean 
citation frequency] was found between 20-25 papers per 
scientist. . .Rather low correlation was found between the 
status of a journal. • .and citation frequency. However, the 
niimber of citations a paper received was found to be highly 
correlated to its rating by a peer review process." 

Lightfield, E. Timothy, "Output and recognition of sociologists," 
The American Sociologist , 6 (May 1971): 129-133. File 
"Citations." COZZENS. 

Quantity of output measured by numbers of papers; quality by 
citations. 

MacRae, Duncan Jr., "Growth and decay curves in scientific 
citations," American Sociological, Review , 34 (Oct. 1969): 
631-635. File "Citations." 

Uses an "exponential model," extending earlier work of 
Price (see Little Science, Big Science , Sec. I.A.), to 
separate the effects of growth of the literature and 
selection practices favoring more recent articles on 
citation. One finding is that "citations in sociology tend 
to refer to older articles than those in the natural 
sciences. " 

MacRoberts, Michael H. and MacRoberts, Barbara R. , "The 
negational reference: Or the art of dissembling," Social Studies 
of science . 14 (1984): 91-94. File "Citations." 

Discusses possible reasons for the unexpected paucity of 
critical references in scientific literature and techniques 
which may be used to criticize implicitly. 

Martin, Ben R. , Irvine, John, and Crouch, David, "Science 
indicators for research policy: A bibliometric analysis of ocean 
currents and protein crystallography," University of Sussex, 
Science Policy Research Unit, Occasional Paper No. 23 (1985) . 
Copy without appendices in File "Martin/Irvine"; complete copy in 
Kennedy School of Government Library, Science Policy Collection. 

Martin and Irvine atteaqpt to extend th^feir methods of 
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evaluating research performance of laboratories or 
institutes (see entries in Sec. 11 under ilartin or Irvine) 
to sub-disciplines within coxintries as a whole. Because of 
the size and cost of the undertaking, the authors abandon 
the use of peer review and rely exclusively on "numbers of 
publications, numbers of citations, and a citation-based 
measure of the scientific 'influence' of research" (p. 14). 

Merton, Robert K. , Foreword to E. Garfield, Citation Indexing; 
Tts Theory and Application in Science ^ Technoloerv. and Humanities 
(N.Y.: John Wiley & Sons, 1979). Cabot Science Library Z697 .S5 
.G37. Reprinted in current Contents . July 9, 1979: 7-10. File 
"Citations." 

Review of citation analysis from the point of view of 
sociology of science. 

Moed, H.F., et al., "The use of quantitative data for the 
measurement of university research perf ormanse , " in W. Callebaut, 
et al., eds., George Sarton Centennial (Ghent, Belgi\imi 
Commionication and Cognition, 1984). File "Citations." 

S\ammary of research on "quantitative, literature-based... 
indicators as tools for university research policy." The 
study was prompted by "a drastic change of the allocation 
system at the University of Leiden." Two faculties within 
the university were evaluated, and the faculty members 
interviewed, but the results are not presented here. 

Moore, William J., "The relative quality of graduate programs in 
economics, 1958-1972: Who published and who perished," Western 
Y-connmic Journal (1973): 1-23. File "Education." 

Uses rather elaborate mathematical models to find that 
publication rates are highly correlated with the Cartter 
report's rankings. See also Knudsen and Vaughan (Sec. I. A) 
and Beyer and Snipper (Sec Vll) . 

Mullins, Nicholas C, Hargens, Lowell L. , et al., "The g^^jup 
structure of co-citation clusters: A comparative study," AJ"erican 
sociological Review . 42 (Aug. 1977): 552-562. File "Citations." 

Uses "block modeling of data from a sociometric 
questionnaire" to "analyze the patterns of social structure 
shown by authors of two highly cocited clusters of 
biological-science papers." The two groups show distinct 
differences. See also Shinn paper in Sec. IX. 

Narin, Francis, T^raluative Bihliometrics; The Use of Publication 
.nrq rkatlon Anal y ^^^s In the Evaluation of Pc entific Activities 
Report by Computer Horizons, Inc., Cherry Hill, N.J., to the NSF 
(1976). Document on microfiche in file "Citations." ABT. 

Abstract from Abt Associates' ^^-thods for the Strategic 
Evaluation nf Research Program s f Rlbllographv) (p. 10; see 

o 16 

ERIC ^^^^ 



11 



introduction) : "This report examines extensively piiblication 
and citation analysis as a means of assessing scientific 
activity. It examines three major aspects of bibliometrics: 
1) its historical development, 2) the numerous empirical 
studies that have been xmdertaJcen on the validity of 
evaluative bibliometrics, and 3) the •influence 
methodology, » or the procedure for calculating individual 
journal influence. , .in summarizing the results of two dozen 
comparative studies, the author reports: 'These studies all 
tend to show that literature- based measures of the quality 
and quantity of scientific output correlate positively with 
non- literature measures. Peer evaluation of the eminence of 
scientists and of scientific institutions are almost always 
correlated with both citation and publication measures... • 
The author's report [also] points to some of the problems 
inherent in evaluative bibliometrics." 

Narin, Francis, et al., "Inter-relationships of scientific 
journals," Journal of the American Society for Information 
Science, 21 (1972): 323-331. Not held in files. COZZENS. 

From Cozzens' abstract: "The patterns of cross-citing among 
275 journals in mathematics, physics, chemistry, 
biochemistry, and biology are 'mapped, • showing clear 
boundaries between fields and sub-fields. . .These maps 
support the intuitive hierarchy of fields: math to physics 
to chemistry to biochemistry to biology." 

Narin, Francis and Carpenter, M.P., "Further development of 
indicators of the quantity and quality of the scientific 
literature," Amendment No. 4 to report NSF-C627 (Cherry Hill, 
N.J.: Computer Horizons, Feb. 1974). Not held in files. 

Narin, Francis and Carpenter, M.P., "National publication and 
citation comparisons," report NSF-627 (Cherry Hill, N.J.: 
Computer Horizons, May 1974); version with same title published 
in Journal of the American Society for Infor mation Science. 2S 
(March-April 1975): 80-93. JASIS version in File "Citations." 

The paper describes the authors' development of citation and 
publication indicators for the NSF's first Science 
Indicators volume (1972) . Compares publication and citation 
counts (per paper) across countries using data from the 
Science Citation Index and abstracting services. 

Price, Derek de S., "The citation cycle," Current Contents. 12 
(Sept. 29, 1980): 8-20. File "Citations." 

Develops a model of scientific progress based on "an 
interlocking metabolic complex of bibliometric (and 
scientometric) parameters in a comprehensive and integrated 
structure after the manner of the Nitrogen Cycle and other such 
paraphernalia beloved of organic chemists and ecologists." See 
also Garfield comments on the model (Sec. I.A) . 
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price, Derek de Solla, "A general theory of bibliometrio and 
other cumulative advantage processes," Journal of the American 
^oclet v for Information Science . 22. (Sept. -Oct. 1976): 292-306. 
File "Citations." COZZENS. 

price develops a relatively elaborate mathematical model for 
evaluating cumulative advantage processes in the scientific 
community, which would in principle use bibliometrio data. 

price, Derek J. de Solla, Little Science. Big Science (N.Y.s 
Columbia Univertsity Press, 1962). Cabot Science Library Q171 
.P9464. COZZENS. 

Cozzens' abstract: "The exponential law of scientific growth 
is demonstrated for several social dimensions. The logistic 
character of the curves for publishing, numbers of Ph.D.'s, 
etc. indicate that the U.S. is at the end of its exponential 
growth period and its rate should begin to decline relative 
to other countries. As Price's emphasis is primarily on 
statistical bibliography, discussion of citation is 
limited..." 

price, Derek J. de Solla, "Networks of scientific papers," 
science, ^49 (July 30, 1965): 510-515. Pile "Citations." 
COZZENS. 

Takes into account the time elapsed between publication of a 
paper and citation of that paper, as well as the nvimbers of 
citations between papers. Uses the data to map, in a 
preliminary way, "research fronts," and suggests that "with 
such a topography [ot research fronts] esteiblished, one 
could perhaps indicate the overlap and relative importance 
of journals and, indeed, of countries, authors, or 
individual papers by the place they occupied within the map, 
and by their strategic centralness within a given strip [or 
portion of a research front]." 

Price, Derek J. de Solla, "The science of scientists," ^eriical, 
opinion and Review , i. No. 10 (1966): 88-91, 94-97. Not in 
files. COZZENS. 

From Cozzens' abstract: "The development of scientific 
laws about the nature of science begins with the recognition 
of the scientific paper as the basic unit of scientific 
structure. . .Because of this, citation patterns can reveal 
aspects of this structure..." 

Price, D.J.d.S. and Gursey, S., "Studies in scientometrics, 1: 
Transience and continuance in scientific authorship," and 
"Studies in scientometrics, 2: The relation between source author 
and cited author populations," Jntemational Forum on Information 
anj_Pocutnentation . 1, No. 2: 17-24; No. 3: 19-22 (1976). Not in 
files. COZZENS. j^Q 

From Cozzens' abstract: "A statistical model is created which 
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describes the basic structure over time of the publishing 
population of scientists. . . [The] nodel. . . is extended. . ,ttnd 
indicates a high correlation between the few prolific 
authors and the few highly cited ones." 

Roche, Thomas and Smith, David Lewis, "Frequency of citations as 
criterion for ranking of departments, journals, and individuals," 
sociological Inquiry . 48/ No. 1 (1978): 49-57. File "Citations." 

Attempts to measure the quality of sociology departments. 
Builds on earlier work which suggests that the Science 
Citation Index would be valuable in performing this task. 
Has a very useful bibliography of the sociological 
literature on citation analysis. 

Schubert, A. and Braun, Tiber, "Some scientometric measures of 
publishing performance for 85 Hungarian research institutes," 
Seientometrics . 3, (Sept. 1981): 379-388. MIT Dewey Library. 

Authors' abstract: "Values of and correlations between some 
measures of publishing performance, scientific manpower, and 
citation impact were compared across" several research 
fields. 

: 

stern, Nancy, "Age and achievement in mathematics: A case study 
in the sociology of science," Social Studies of Science . 1 
(1978): 127-140. File "Citations." 

Provides evidence that citations are a "crude indicator of 
quality of work in mathematics," but that "age explains very 
little, if anything, about productivity" or quality of work 
(i.e. young mathematicians are not more likely to produce 
highly cited papers) . 

Vlachy, Jan, "Physics journals typology, group rankings by 
citation and immediacy," Czechoslovak Journal o f Phvsics B. 15 
(1985): 589-592. File "Citations." 

These and other entries for Vlachy are brief essays whose 
titles are self-explanatory. 

Vlachy, Jan, "Priority choice and research front specialties in 
physics," Czechoslovak Journal of Phvsics B . 14 (1984): 95-98. 
File "Citations." 

"Vlachy, Jan, "Publication image of European physics," 
Czechoslovak Journal of Phvsics B . 34. (1984): 891-594. File 
"Citations . " 

Vlachy, Jan, "Research fronts in physics 1983," Czechoslovak 
Journal -of Phvsics B . M (1984): 171-174. File "Citations." 

Vlachy, Jan, ed. , "Scientometric Analysis in Physics," special issue of 
the Czechoslovak Journal of Phvsics . B 36 . No. 1 (1986) . Shelf. 
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Forty-six papers on the evaluation of science, most of which 
focus on bibliometric methods. The authors, from 13 
countries, include many of the world's experts on the 
evaluation of research. Vlachy offers an introduction and 
guide to the volume, with 186 references. A subject index 
is included. 

Vlachy, Jan. "World physics pxiblication output — country 
distribution and trends," Czechoslovak Journal of Phvsics B . 
35 (1985): 705-708. File "Citations." 

White, Howard D. and Griffith, Belver C. , "Author cocitation: A 
literature measure of intellectual structure," Journal of the 
American Society for Infoinotiation Science . 32 (May 1981) : 163-171. 
File "Citations." 

From the authors' abstract: "It is shown that the mapping of 
a particular area of a science. . .can be done using authors 
as units of analysis and the cocitations of pairs of authors 
as the variable that indicates their 'distances' from each 
other. The analysis assiames that the more two authors are 
cited together, the closer the relationship between them." 

Zunde, P. and Slamecka, V., "predictive models of scientific 
progress," Information Storage and Retrieval . 7 (1971): 103-109. 
Not in files. COZZENS. 

From Cozzens' abstract: "The authors describe the use of Markov 
chain probability analysis to develop a model of long-run 
scientific development..." using in part citation patterns. 
"The model predicts... a general movement of activity from 
science and technology to social science disciplines." 



B. Critiques of citation analysis 

Bavelas, Janet Beavin, "The social psychology ot citations," 
Canadian Psychological Review . 19 (April 1978): 158-163. File 
"Critiques of Citation Analysis." 

suggests that scholars cite other scholars "both because of 
scholarly impact and also... to show that we Icnow the 
pertinent works in the area." Thus, citation counts may 
"measure social consensus as well as, and indistinguishably 
from, scholarly impact." The paper is basically critical of 
quantitative measures of quality of research,, but advocates 
thinking for oneself rather than a complete return to the 
'old boy network.' 

Borenius, G. and Schwartz, S., "Remarks on the use of citation 
data in predictive models of scientific activity," Information 
Storage and Retrieval . (1972): 171-172. Not in files. 
COZZENS. 20 

From Cozzens' abstract: "...examines the assumptions 
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necessary to the use of the Markov chain predictive Bodel of 
scientific activity.** (See Ziinde and Slamecka, Sec. I. A.) 
The authors discuss pxroblems vith this approach, which 
relies on certain forms of citation analysis. 

Brooks, Terrence A., "Private acts and public objects: An 
investigation of citer motivations," Journal of the American Society 
for Information Science , 36 (July 1985): 223-229. File "Citations." 

Empirical study of citer motivations. Finds that authors in 
both the humanities and the sciences used citations largely 
to "persuade," though the patterns of motivations between 
the two sets of disciplines varies greatly. "This is 
preliminary evidence that citer motivations cannot be 
treated as an undifferentiated whole; the specific rankings 
of citer motivations may differ by subject area. Less 
tenable now are simple euid rational motivational models 
[such as those of Price, et al.; see references p. 227]." 

Chubin, Daryl, "On the use of the Science Citation Index in 
sociology," American Sociologist , 8. (November 1973): 257-267. 
File "Critiques of Citation Analysis." COZZENS. 

Although Chubin feels that citation analysis as such 
is a useful tool for sociologists in assessing the quality 
of research, he argues that sole reliance on the SCI may 
result in badly biased data, especially for fields "like 
sociology." He concludes that the main problems with 
the SCI are that it indexes only recent journals in 
sociology (but recall that this article was written in 
1973), it does not account for widespread collaboration in 
the field, and it does not index second, and usually junior, 
authors . 

Cole, Jonathan and Cole, Stephen, "Measuring the quality of 
sociological research: Problems in the use of the Science 
Citation Index , American Sociologist , 6 (February 1971): 23-29. 
File "Critiques of Citation Analysis." COZZENS. 

The Coles discuss 9 potential problems with the SCI but 
conclude that "Nevertheless, the value of using them 
[citation countF] as rough indicators of the quality of a 
scientist's work should not be overlooked." 

Duston, B. and Corbett, J.W., "Citation index mixleading," 
letter, Phvsics Today , 29 . No. 9 (Sept. 1976): 89. File 
"Critiques of Citation Analysis." 

Points out that different fields of physical science 
(and mathematics) have different citing norms, making 
comparisons between scientists in different fields 
problematic. 

Endler, Norman S., "Beyond citation counts: Developing research 
profiles," Canadian PsVcholeolcal Review , 2JL (April 1978): 
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152-157. File "Measuring Output." 

Author's abstract: "Recent studies using... SCI publication 
and citation counts as indices of productivity and scholarly 
impact are reviewed. The advantages and limitations of this 
method of asessing scholarship were discussed. Finally, a 
more broadly based research profile was outlined." The 
profile weights various types of publications and addresses. 

Fox, Mary Frank, "Publication productivity among scientists: A 
critical review," social studies of Science . 13 (1983): 285-305. 
File "Critiques of Citation Analysis." ABT. 

A review of literature on scientific productivity, written 
by a sociologist. Discusses three types of explanations for 
variations in productivity: 1/ individual-level 
characteristics; 2/ environmental location; and 3/ feedback 
process of cumulative advantage and reinforcement. 

Garfield, Eugene, "Uses and misuses of citation frequency," 
Current Contents (October 28, 1985): 3-9. File "Citations." 

Detailed analysis of the topic. 

Gilbert, G. Nigel, "Referencing as persuasion," Social Studies of 
S cience . 2, No. 1 (Feb. 1977): 113-122. File "Sociology of 
Science." COZZENS. 

The author considers "scientific papers as 'tools of 
persuasion.' A scientist who has obtained results which he 
beleives to be true and important has to persuade the 
scientific community ... to share his opinions of the value of 
his work. For it is only when some degree of consensus has 
been achieved that his research findings will be transformed 
into scientific knowledge." 

Lindsey, Duncan, "Production and citation measures in the 
sociology of science: The problem of multiple authorship," Social 
Studies of science . M (May 1980): 145-162. File "Critiques of 
Citation Analysis." 

Concludes that neglect of multiple authorship in assessing 
research output introduces "intolerable error" and "often 
profoundly influence (s) substantive interpretation." 

Long, J. Scott, et al., "The problem of junior-author papers in 
constructing citation counts," social Studies of Science, 10 (May 
1980): 127-43. File "Critiques of Citation Analysis." 

Investigates in some detail the methodological pitfalls 
associated with using citation indices which Ixst only first 
authors . 2 2 

Moravcsik, Michael J., "Measures of scientific growth," Research 
Policy . No. 3 (Oct. 1973): 266-276. File "Critiques of 
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Citation Analysis." COZZENS. 

"Nvunbers of publications and citation ratings have recently 
been used as measures of scientific growth. The present 
paper discusses a number of presumed weaknesses of these 
measures. First, a distinction is made among scientifc 
activity, scientific productivity, and scientific progress. 
Then it is suggested that the above measures might depend on 
the particular field of science, on the speed whereby 
research front Information becomes archival, on the 
phenomenon of wrong papers and 'also ran papers, on the 
geographical differences in communication patterns, on 
whether we want to measure activity, productivity, or 
progress, and on the temporal variations in scientific 
communication patterns. Though some examples are given, the 
quantitative substantiation of the proposed effects must 
await further research" 

Horavcsik, Michael J. and Murugesan, P. , "Some results on the 
function and quality of citations," Social Studies of Science ^ 5 
(1975): 86-92. Not in files. COZZENS. 

From Cozzens' abstract: "...references from. . .physics 
articles. . .are categorized as: conceptual ' or operational, 
organic or perfunctory, evolutionary or juxtapositional, and 
confirmative or negational. . .The large number of perfunctory 
citations (41%) raises serious doubts about the use of 
citation as a quality measure..." See also the literature 
review on citation context analysis by H. Small in Sec. I.C. 

Herman, Edward T., "Citation analysis and the current debate over 
quantitative methods in the social studies of science," 4S 
(Society for Social Studies of Science) Newsletter , 5 (Summer 
1980): 7-13. File "Critiques of Citation Analysis." 

Intellectual history of the debate over the appropriateness 
of citation and co-citation analysis in the history of 
science. Focusses on Price, Garfield, and the Coles on the 
one hand, and Edge (see Sec. VI.) on the other (though other 
authors are discussed.) Short but very useful set of 
references. 

Porter, Alan L. , "Citation analysis: Queries and caveats," Social 
Studies of Science , 7, No. 2 (May 1977): 257-267. File 
"Critiques of Citation Analysis." COZZENS. 

Review of the value of citations and methodological 
problems with their use. 

Pragier, G. and Ronayne, J., "Criticism of the use of citation 
analysis in studying the science-technolocy relationship," 
Journal of Chemical Information and Computer Science . 15 (1975) : 
155-157. Not in files. COZZENS. 

The authors discuss problems with assessing the effect of 
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academic research on industrial innovation using citation 
analysis. 

Satariano, William A., "Journal use in sociology: Citation 
analysis versus readership patterns," Library Quarterly. 48 
(1978): 292-300. File "Measuring Output." 

Author's abstract: "This paper examines the hypothesis that 
citation patterns parallel readership patterns in 
sociology." He finds that "[C]itation patterns reflect a 
cross-disciplinary focus that is not found in the journals 
most often read. Further, citation studies underestimate 
the usefulness of popular social science periodicals and 
specialty and regional journals in sociology." 

Thome, Frederick C. , "The Citation Index: another case of 
apurious validity," Journal of Clinical Psychology . 21 (October 
1977): 1157-1161. File "Critiques of Citation Analysis." 

An impressionistic account of a number of what the author 
sees as severe problems with citation analysis. Citation 
analysis itself is termed "One of the most amazing 
pseudoscientific popularity contests..." 



C. Citation context analysis 

Chubin, Daryl E. and Moitra, Soumyo D. , "Content analysis of 
references: Adjunct or alternative to citation counting?" Social 
Studies of science . No. 4 (Nov. 1975): 423-440. File 
"Content analysis." 

Studies the uses and limitations of citation analysis 
and the merits of another, more qualitative approach to 
evaluating the quality of research: content analysis. 

Schubert, A., Zsindely, S., Teles, A., and Braun, T. , 
^Quantitative analysis of a visible tip of the peer review 
iceberg: Book reviews in chemistry," Se;Lentometrics . 6 (Nov. 
1984): 433-443. MIT Dewey Library. 

Authors' abstract: "Book reviews in chemistry are 
practically unique in being piiblic, 'visible' manifestations 
of the peer review process. Two hundred reviews of 39 books 
on chemical topics were subjected to statistical context 
analysis. Dominance of attitudes, consensus among 
reviewers, correlation between the reviewers' evaluations 
and the stabseguent citation rate of the reviewed book were 
analysed. " 

Small, Henry, "Citation context analysis," in B. Dervin and M.J. 
Voigt, eds.. Progress tn Commu nication Sciences. Vol. Ill, 
(Norwood, N.J.: ABLEX, 1982): 287-310. 

A literature review of work on citation context analysis by 
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researchers dissatisfied with citation counting as such, 
either for the puiTDose of information retrieval or scholarly 
sociology of science. Small examines the work of Moravcsik, 
Chxibln, Cole (see Small's references, some of which are 
included here in sees. I and II) and others. 
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II. Qualitative aipproaches to and more general works on research 
evaluation 



Abt Associates, "An approach to the evaluation of basic research 
projects," Vol. I, report to the National Science Foundation 
(Cambridge, Mass.: Abt, June 1967). Shelf. ABT. 

Authors' abstract: "This report describes a study of the 
feasibility of evaluating basic research projects. A method 
was developed to assist with the evaluation of coiupleted 
py-Q^eGts In terms of the ir impact within the scientific 
community . Measures were developed to assess the impact of 
completed projects on scientific research areas by examining 
characteristics of use , patterns of diffusion, and 
tertllitv . Additional measures were proposed that could be 
applied to Interrelated groups of projects (fields of 
scientific activity) that would indicate characteristics of 
activity, connectivity, and advancement. 

"A method was suggested to assist with the evaluation of 
basic research proposals in terms of the. expected 
scientific impact. The proposals would related to a 
structure of questions of current resea Interest and 
evaluated in terms of the degree to whlc> c ^^so questions 
might be answered, the degree to which th^y might be 
changed, and the extent to which the proposal might 
Influence the field as a whole by addressing questions of 
wide Interest. Because both funded and unfunded proposals 
could be measured, the development of this method would also 
present the possibility of estimating scientific 
opportunities missed due to resources insufficient to 
support all meritorious proposals." (Emphasis in the 
original. ) 

Abt Associates, "A comparative study of the prospective and 
retrospective approaches to the evaluation of proposed basic 
research," report to the National Science Foundation (Cambridge, 
Mass.; Abt, July 1969). Shelf. ABT. 

Investigates "the possibility of developing measures that 
could be used in an operationally realistic way to 
anticipate future effectiveness of present investment 
alternatives in basic research." "...measures are based on 
changes in basic knowledge proposed by researchers seeking 
financial support, and the use made by siabsequent 
investigators of knowledge gained from completed projects." 
An ambitious attempt to precisely specify potential and 
realized advances in knowledge. 

Abt Associates, "A methodological approach to measuring change in 
science produced by basic research," report to the National 
Science Foundation (Cambridge, Mass.; Dec. 1968). Shelf. ABT. 

Together with the companion volumes listed immediately 
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above, this is an attempt to systematically evaluate and 
predict the impact of basic research projects. From the 
authors • eibstract: "This study has been devoted to the 
development of a methodology itor characterizing and 
measuring chemge in scientific Jcnowledge produced by 
individual research projects. The final system which has 
emerged consists of three basic components: 1. A 'language' 
which permits the classification and description of -the 
fundamental relations in a field of science. 2. A set of 
'parameters of change' which identify the 'dimensions' in 
which a change in the fundeumental relations can occur. 3 . A 
mathematical formula for scaling the Range-Precision 
dimension of change. 

"The 'language' and 'parameter£i of change' together form the 
Qualitative Theory of Scientific caiange. The mathematical 
formulation represents the first step in developing a 
Quantitative Theory of Scientific Change" (emphasis added) . 

s 

Abt Associates, "Methods for the strategic evaluation of research 
programs: The State-of-the-Art, " report to the National Science 
Foundation (Caunbridge, Mass. : 1985) . Shelf. 

Based primarily on a thorough literature review. Addresses 
both governmental (intramural and extramural) and industrial 
research, and attempts to answer the following questions: 
"1. How is evaluation generally done in the organization 
under consideration (viz., government or industry 
laboratory, funding agency)?... 2. In what ways, if any, 
are various evaluation techniques combined? 3. On which 
aspects of research outcomes do the evaluation procedures 
focus? 4. Und-:r what circumstances are quantitative 
techniques used nos* often? C^mphasis added] 5. What is 
known and thought about the effectiveness of the various 
techniques in use?.. 6. What kinds of available techniques 
have nafe been tried and why? ... 7 . What is known about the 
impaest of evaluation procedures on planning and resource 
allocation in each type of organization or fianding program? 
8. What further research is needed to improve methods 
presently used?" In answer to question 1, the following 
approaches to evaluation are studied: peer judgement, 
L^ocio-psychological measures, bibliometric analyses, 
econometric and related methods, restrospective analysis, 
and multidimensional methods. This report is accompanied by 
an annotated bibliography (see Introdution to the present 
bibliography) . 

Andrews, Frank M. , ed. , Scientific Productivity: The 
Effectl "^ eness of Research Groups in Six Countries (Cambridge, 
England and Paris: Cambridge University Press and UNESCO, 1979). 
Shelf. r^BT. 

A report on an international study coordinated by UNESCO and 
performed by local researchers in each of six European 
countries: Austria, Belgiim, Finland, Hungary, Poland, and 
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Sweden. The purposes of the book are to "find ways to 
enhance the performance effectiveness of research units and 
their members" and "to develop and test methods for 
assessing the organization and performance of research 
iinits" (p. xxix) . See also review by Blume (Sec. II.). 

Blume, Stuart S., "A managerial view of research," Science. g07 
(January 4, 1980): 48-49. File "Measuring Output." 

A largely negative review of Andrews, ed. , Scientific 
Productivity (Sec. IX). Blume's fundamental criticism is 
that "Most of the contributions. . .adopt a perspective 
derived in some way from organizational theory and tev..r, to 
look at research groups as more or less isolated formal 
organizations." The sociology of science, he feels, offers 
more insight into the "production (and validation) of 
scientific knowledge." 

Boggio, Gr and Gallimore, R., eds. , Evaluatio n of Research and 
Development! Methods for Evaluating the Res ults of European 
Community R&D Programs - (Dordrecht, The Netherlands and Boston: D. 
Reidel, 1982). Table of contents in File "Measuring Output"; 
book Widener L.C. Q.180 .E9 E93 1982. 

The 10 papers in this collection were presented at a 
conference sponsored by the Commission of the European 
Communities. They provide valuable insight into the content 
of the EC's R&D programs, but, contrary to the intent 
suggested by the title, do not shed much light on methods 
for assessing research. 

Bo-M-gio, G. and Spachis-Papazois , Evaluatio n of Research and 
Development; Methodologies for RSD Evaluat ion in the European 
Community Member states, the United States of America, and Japan 
(Dordrecht, The Netherlands and Boston: D. Reidel, 1984). Baker 
Library T177 .E9 E92 1984. 

Bradshaw, Gary F. , et al., "Studying scientific discovery by 
computer simulation," Science . £22 (Dec. 2, 1983): 971-975. File 
"Measuring Output." 

Authors' summary: "BACON is a computer program that 
simulates some of the i^nportant processes of scientific 
discovery. When provided with data €ibout temperatures 
before and after two substances are brought into contact, 
the program infers the concept of specific heat and arrives 
at Black's law of temperature equilibrium. Comparison of 
BACON'S discovery methods with historical records of Black's 
work casts light upon the relations between data-driven and 
theory-driven discovery . " 

Bruer, John, "The search for quality information: Schistosomiasis 
literature," in K.S. Warren, ed., Selectivity in Information 
systems : 144-153. (See below, Sec. II.) 
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An attempt to develop Indicators of quality of research 
which can be used with a computerized retrieval service, 
using schistosouiasis literature as an exeunple. Bruer 
identifies six indicators which can be used with Index 
Medieus or Science Citation Index . These are all either 
scalar or binary. (Examples of binary indicators are 1/ 
published in a "highly cited journal" and 2/ "written in 
English") . 

Cetron, Marvin J. and Martino, Joseph, "The selection of R&D 
program content — survey of quantitative methods," TEEE 
Transactions on Engineering Management . EM-14 . No. 1 (March 
1967). File "Measuring Output." ABT. 

Authors' zJDstract: "This paper presents a summary of methods 
of evaluating and selecting R&D projects. Approximately 
'^;hirty methods, which have appeared in scattered places in 
the literature, are described briefly, and a bibliography is 
provided for further information. The various methods are 
coiapared and contrasted with each other relative to a 
standard set of characteristics relating to ease of use, and 
to scientific or technological area of applicablity . " 

Chang, K.H., "Evaluation and survey of a subfield of physics: 
Magnetic resonance and relaxation studies in The Netherlands," 
(Utrecht, The Netherlands: Stichting F.O.M. , report no. 
FOM-37175, 1975) . Shelf. 

Author's abstract: "A survey of the Dutch activities ... and 
the significant publications in the sxibfield [are 
presented] ... it is concluded that the various approaches 
which have been followed to evaluate the impact of the 
particular studies, viz. interviews with experts, scanning 
of textbooks, searching for awards, analyzing the Science 
Citation Index, etc., have yielded non-contradictory 
results... A new procedure to analyze data obtained from the 
Science Citation Index is discussed in the appendix." 
Includes a section in the concluding chapter entitled 
"Evaluation: citations, a measure for quality?" 

Comroe, Julius H. , Jr. and Dripps, Robeirt D. , "Scientific basis 
for the support of biomedical science," Science, 192 (April 9, 
1976) : 105-111, File "Disciplinary" ; also in H.H. Fudenberg and 
V.L. Melnick, eds.. Biomedical Scientists and Public Policy 
(N.Y.: Plenum Press, 1978): 15-34. Widener L.C. R.854 U5 B53. 

A summary of the authors' NIH project, immediately 
below. See Garfield entry, "How can we prove the value of 
basic research?'* (Sec. II.) for discussion of Comroe and 
Dripps' paper. 

Comroe, Julius H., Jr. and Dripps, Robert D. , The Ten Top 
Clinical Advances in Cardlo-vascular-taulmonarv Medicine. 
1945-1975 . 2 Vols., report to the National Institutes of Health, 
NIH 78-1522 (Washington, D.C.: Dept. of Health, Education and 
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Welfare, 1977) . 

The authors asked 100 specialists to vote on the top 
advances. They then identified "essential bodies of 
knowledge" necessary for each. Finally, they attempted to 
identify core published papers which advanced these areas of 
knowledge. They read many of these papers and classified 
them as basic, applied, or clinical research; development or 
engineering; or review papers. The authors found that basic 
research was fundamental to almost twice as many discoveries 
as all of the other categories combined, euid that it should 
therefore be funded more generously. The project was 
inspired in part by dissatisfaction with Project Hindsight, 
a restrospective analysis of military research (see Sherwin 
and Isenson below. Sec. II) . Compare also TRACES project, 
listed in Sec. II under Illinois Institute of Technology. 



Cook, Thomas D. and Leviton, Laura C, "Reviewing the literature: 
A comparison of traditional methods with meta-analysis," Journal 
of Personality . M (Dec. 1980): 449-472. File "Measuring 
output." 

An investigation of the relative merits of meta-analysis 
("the name given to a set of techniques for reviewing 
research in which the data from different studies are 
statistically combined") and "traditional," or qualitative, 
literature reviews. The authors defend the merits of the 
latter and point out "unique limitations" of the former. 
This paper provides important background for the debate over 
bibliometric methods in the evaluation of scientific 
research. See also entry under Light and Pillemer (Sec. 
II.) . 

coward, H. Roberts and Franklin, J. Jeffrey, "Setting priorities 
for science-driven technology: Patents and bibliometric models, 
in Carole Ganz, ed.. The Rise a nd Fall of Priority Fields, U.S. 
papers presented at a National Science Foundation/European 
Science Research Councils Symposium, Paris, France, Sept. 22-24, 
1985. Shelf. 

Investigates the relationship between basic research and 
technological innovation using co-citation maps to represent 
f '.elds within the former and patent data to represent the 
latter. Focuses on solid state physics and concludes that 
"the strong pattern of interaction between the bibliometric 
model and patenting activity in solid state devices suggests 
that the technique might be used in order to identify 
research-driven technologies and technology-driving research 
areas. .. [This] is increasingly important for research 
priority setting." 

Endler, Korman S., "Beyond citation °o«nts: Developing research 
profil4s," Canadian Pp vcholoaical Review, 19 (April 1978). 
152-157. File "Measuring Output." 
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Author's abstract: "Recent studies using... SCI publication 
and citation counts as indices of productivity and scholarly 
impact are revi^^wted. The advantages and limitations of this 
method of asessing scholarship were discussed. Finally, a 
more broadly based research profile was outlined." The 
profile weights various types of publications and addresses. 

Freeman, Christopher, "Measurement of output of research and 
experimental development: A review paper" (Paris: UNESCO, ST/S/16 
COM.69/XVI-16A, 1970). File "Measuring Output." 

Review of the basic theoretical and practical problems 
involved with measuring research output* Includes 
discussions of research publications, patent statistics, and 
"innovations" as measures of productivity. 

Fudenberg, H. Hugh, "Informing the public: Fiscal returns of 
biomedical research," in H.H. Fudenberg and V.L. Melnick, eds., 
Biomedical Scientists and Public Policy (N.Y.: Plenum Press, 
1978): 35-48. Widener L.C. R.854 U5 B53. 

Argues that basic research in biomedicine yields 
identifiable (after the fact) financial savings. 

Garfield, Eugene, "How can we prove the value of basic research?" 
Current Contents (October 1, 1979). File "Citations." 

Cites favorably and summarizes the arguments from two papers 
in a volume on biomedical research by Fudenberg and Melnick. 
One paper is by Comroe and Dripps (see entry in Sec. II.) 
and attempts to develop a "more objective justification for 
supporting basic research," particularly in biomedicine, 
than had previously been offered. Comroe and Dripps do so 
by tracing discoveries back to seminal papers and then 
determining whether these papers represented basic or 
applied research. Garfield also discusses a paper by 
Fudenberg (Sec. II.) which argues that basic research in 
biomedicine yields identifiable (after the fact) financial 
savings . 

Gilbert, G. Nigel, "The transformation of research findings into 
scientific knowledge," Social Studies of Science , 6 (1976): 
281-306. File "Sociology of Science." COZZENS. 

The paper is concerned with "the process whereby a 
scientist's research findings are transformed into 
accredited factual knowledge. .with the 'context of 
justification'... In contrast to the epistemological concerns 
of the philosophers^" the author considers "only the 
procedures actually used by the natural scientists to decide 
on the validity of claims to scientific knowledge." 

Giordano, Richard, "Using computers for text analysis: The 
development of GATOR," center for the Social Sciences Newsletter 
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(Columbia University) , No. 2 (Spring 1985): 3-6. File 
"Measuring Output." 

Discusses the text analysis software developed for Cole and 
Zuckerman's study of men and women scientists (see Sec. XI), 
in which they used both quantitative and qualitative 
(interview) data. 

Griffith, Belver C. , ed. , ^ey Pacers in Information Science 
(White Plains, N.Y. : Knowledge industry Publications for the 
American Society for Information Science, 1980). Widener L.C. 
Z.699 .K422. 

Includes papers on scientific communication (D. Crane, S. 
Crawford, B. Griffith), publication and citations (F. Narin, 
D. de Sella Price, H. Small) and other topics more directly 
pertinent to information retrieval and librarianship. 

Goffman, William and Warren, Kenneth S., Scientific information 
Systems and the Prinr rjpTe of Selectivity (N.Y.: Praeger, 1980). 
Cabot Library Z. 699.5 S3 G63. 

This project is motivated by a desire to better manage the 
medical literature, and draws upon expertise in both 
mathematical library science and epidemiology. (The growth 
of information is modelled by mathematics which is applied 
to the spread of disease.) Chap. 5 is entitled "Qualitative 
assessment of the scientific literature." The authors note 
here that "A complete investigation of a medical or 
scientific literature must involve its evaluation on a 
qualitative basis" and that "The best approach for 
qualitative selection of a literature was to obtain a 
consnsus of opinion of a large number of the best and most 
knowledgeable scientists in the field at the present time" 
(p. 117). 

Gottfredson, S.D., et al., "Quality indicators in the scientific 
iournal article pxablicatd.on process," JHU-CRSC report #29 
(Baltimore: Johns Hcpkir/S University, Center for Research in 
Scientific Communication, January 1977). Not held in files. No 
eOsstract available. 

Griffith, B.C., Sm'^11, H.G. , et al., "The structure of scientific 
literatures III Toward a macro- and micro-structure for science," 
Science_StudiSm, A (1974): 339-364. MIT Resource Sharing 
Center. (Not held in files; Part I under Small.) 

Griliches. Zvi, "Economic problems of measuring returns on 
research!" in Elkana, et al., eds.. Toward a Metric of Sc;Lence 
(N.Y.: Wiley, 1978): 171-177. Shelf. 

Focuses on patents, invention lists, and publication counts. 

eriliches Zvi. ed., Pfcn. Patents, and Productivity (Chicago: 
uSie«??y Of Chfc^go Pr;ss, 1984). See especially tha following 
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papers: Pakes, Ariel and Griliches, Zvi, "Patents and R&D at the 
firm level: A first look," pp. 55-71; Griliches, Zvi, "Market 
value, R&D, and patents," pp. 249*258; Griliches, Zvi and 
Lichtenberg, Frank, "R&D and productivity growth at the industry 
level: Is there still a relationship?" pp. 465'-501. Papers in 
File "Measuring Output" under griliches. Book in Littauer 
HD30.42 .U5 R2 1984. 

Research on the use of patent statistics to operationalize 
"advances in knowledge" (p. 55) and to evaluate the effect 
of these advances on economic performance. 

Holden, Constance, "Sociology stir at Harvard: Controversial 
tenure decision complicates attempts to bring in more 
•quantifiers'," Science . 228' (Mav 10, 1985): 692-93. File 
"Education. " 

Focuses on the case of Paul Starr, . a "qualitative" 
sociologist refused tenure, but cilscusses the importance of 
quantification in the social sciences. 

Hoi ton, Gerald, "Can science be measured?" in Elkana, et al., 
eds.. Toward a Metric of Science (N.Y.; Wiley, 1978): 39-68. 
Shelf. 

Concentrates "on some fundamental epistemological problems 
adhering to any attempt to make 'indicators,* no matter for 
what puirpose," referring to Science Indicators 1972 and 
other publications for examples. Discusses the problems 
associated with generating quantitative indicators from 
qualitative data, and the potential for using more 
qualitative approaches (e.g. case studies) in evaluating the 
health of science. 

Illinois Institute of Technology, Technolcoy in Retrospect and 
critical Events in Science ("T.R.A.C.E.S.") , report to the 
National Science Foundation, NSF-C535 (Chicago: Illinois 
Institute of Technology Research Institute, 1968). 

A study of the evaluation of the quality of scientific 
research* The authors begin by identifying an important 
scientific achievement, then "trace" the events, research 
programs, and discoveries which were temporally and causally 
prior to that achievement. The authors thereby illuminate 
the mechanics whereby research of high qon^lity is produced, 
with the hope that these mechanics can be applied in various 
research sett^ings. 

Irvine, John and Martin, Ben R. , "Assessing basic research: The 
case of the Isaac Newton Telescope," Social Studies of Science , 
13 (1983): 49-^86. File «Mart in/ Irvine. " (See also entries under 
Martin.) 

Applies the authors' method of "converging partial 
indicators" (see Martin & Irvine, Sec. II) "to evaluate the 
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scientific performance of various optical telescopes." 

Irvine, John and Martin, Ben R. , "Quantitative Science Policy 
Research," unpxiblished paper prepared for presentation to the 
Task Force on Science Policy of the House Committee on Science 
and Technology, October 30, 1985. Includes transcript of 
committee members' questions and Martin's answers. File 
"Martin/Irvine." 

Essentially a review of M&I's research program. Q&A's 
provide some unusual anecdotal insight into M&I's research 
and into the proceedings of the committee. (See also U.S. 
Congress, House Cozomittee . . . , Sec. II.) 

Jones, P.M.S. and Willett, A.L., "Evaluation of the benefits of 
laboratory research and information services," Research Policy, 6 
(1977): 152-153. File "Measuring Output. 

The authors are part of the Programmes Analysis Unit in the 
British Government and have considereible experience with 
evaluation of laboratory research. The paper focuses on 
problems with evaluation associated with government research 
in support of industry; i.e., with ascertaining the 
effectivness of research in assisting industry to achieve 
particular objectives. 

Kevles, Daniel J., "The health of science," Harpers (Aug. 1979): 
26-31. File "Quality in Philosophy." 

Argues that the period 1945-1970 in American science was 
anomalous, that American science is healthy, current 
declining budgets and the warnings of eminent scientists 
notwithstanding, and that continued emphasis on supporting 
the best students and researchers will keep it strong. See 
also Nye (Sec. VI) . 

Kochen, Manfred, "Models of scientific output," in Elkana, et 
al., eds.. Toward a Metric of Science (N.Y.: Wiley, 1978): 
97-136. Shelf. 

The paper attempts "To clarify three conceptual issues 
central to evaluating. . .information systems [to help in 
planning science] ; To compare five models for S&T and S&T 
planning; To evaluate various output measures for the five 
models; To suggest priorities for future research in this 
area and for constructing new science indicators." 

Kuppers, Gunter, Ulitzka, Norbert, and Weingart, Peter, "The 
awarding of the Nobel Prize: Decisions about significance in 
science," in C.G. Bemhard, et al., eds.. Science and Society in 
iihe Time of Alfred Nobel (Oxford: Pergamon Press for the Nobel 
Foundation, 1982). File "Measuring Output." 

"Investigates the structure and pattern of the nominating 
process and its interrelationship with prize decisions." 
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Illustrates with two cases of "revolutionary ideas" in 
physics: quantum theory and relativity theory. Cf, piece by 
Inhaber and Przednowek (Sec. I. A) which uses quantitative 
data in studying the prestige of Nobel winners- 

Light, Richard J. and Pillemer, David B., Suimnlna Up; The Science 
of Reviewincy Research (Cambridge, Mass.: Harvard University 
Press, 1984). Shelf. 

Studies literature and research reviewing. C2iap. 4, 
"Numbers and narrative" (pp. 104-143), is an account of the 
relative merits of qualitative and quantitative approaches 
to reviewing research. See also paper by Cook and Leviton 
(Sex;. II.) . 

Ling, James G. and Hand, Mary Ann, "Federal funding in materials 
research," Science , 209 (Sept. 12, 1980); 1203-1207. File 
"Measuring Output." ABT- 

"The performance of 20 materials research laboratories 
(MRL's) at universities funded with institutional grants 
by. . . (NSF, DOE, and NASA) is evaluated in comparison with 15 
other universities. . .receiving individually funded projects 
for materials research. Performance is measured by peer 
review and citation frequency analysis of publications, 
subjective evalutation of research achievements and 
researcher reputation by a panel of experts, review of 
equipment purchases and utilization, and analysis of 
administrative costs." Finds that productivity is roughly 
the ssone in certain respects, though significant differences 
exist in favor of MRL's. 

Logsdon, John M. , et al., "An c^erview of federal research 
evaluation activities," report prepared by the Program of Policy 
Studies in Science and Technology, The George Washington 
University, for the Division of Policy research and Analysis, NSF 
(April 1985) . Shelf. 

Main chapters are entitled: Defining research evaluation; 
Brief agency descriptions; Specific evaluation approaches. 
Covers all U.S. Government agencies which perform any 
significant amount of research. 

Lowrance, William W. , "The NAS surveys of fundamental research 
1962-1974, in retrospect," Science , 197 (Sept. 23, 1977): 
1254-1260. File "Measuring Output." 

Discusses strengths and veaJcnesses of the ten disciplinary^ 
surveys sponsored by the NAS Committee on Science and Pviblic 
Policy. 

Martin, Ben R. and Irvine, John, "Assessing basic research: Some 
partial indicators of scientific progress in radio astronomy," 
Research Policy , 12 (1983): 61-90. File "Martin/Irvine." ABT. 
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An influential work in the field of research evaluation, the 
policy implications of which have been somewhat 
controversial. The paper develops a method of evaluating 
research "incorporating the following elements: (1) the 
indicators are applied to research groups rather than 
individual scientists; (2) the indicators based on citations 
are seen as reflecting the impact, rather than the quality 
or importance, of the research work; (3) a range of 
indicators are employed, each of which focuoses on different 
aspects of a group's performance; (4) the indicators are 
applied to matched groups, comparing 'like' with 'like' as 
far as is possible; (5) because of the imperfect or partial 
nature of the indicators, only in those cases where they 
yield convergent results can it be assumed that the 
influence [on the indicators] of the "other factors' [not 
related to the "magnitude of the particular (scientific) 
contribution'] has been kept relatively small (i.e. the 
matching of the groups has been largely successful) , and 
that the . indicators therefore provide a reasonably reliable 
estimate of the contribution to scientific progress made by 
different research groups." (Emphasis added.) 

Martin, Ben R. and Irvine, John, "CERN: Past performance and 
future prospects," Research Policy . 13 (1984): 183-210. File 
"Mart in/ Irvine." See also entry under Irvine and Martin. 

First of a series of three papers evaluating the scientific 
performance of CERN. Uses the method of "converging partial 
indicators" (see Martin and Irvine, immediately above) to 
assess the "position of CERN accelerators in world 
high-energy physics relative to those at other large 
laboratories working in the field." This first paper 
evaluates the CERN complex as a whole, while the second 
focuses on individual accelerators within CERN, and the 
third attempts "to assess the future prospects for CERN." 

Martin, Ben R. and Irvine, John, "Internal criteria for 
scientific choice: An evaluation of research in high-energy 
physics using electron accelerators," Minerva. i9 (Autumn 1981): 
408-432. Pile "Martin/ Irvine . " See also entry xinder Irvine and 
Martin. 

A more discursive analysis of the need for more systematic 
methods of research evaluation than the articles on CERN 
listed immediately above. 

Mazlish, Bruce, "The quality of 'The quality of science': An 
evaluation," in La Follette, ed. , Quality in Science (Cambridge, 
Mass.: MIT Press, 1982): 48-67. Shelf. 

suggests that science indicators are inherently value-laden; 
that quantitative indicators deal primarily with factors 
intrinsic to science and "can block other modes of 
tanderstanding"; "that historical analogies. . .and case 
studies are alternate modes of understanding the quality of 
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science..."; that science is, in part, a political process 
involving choices about fundamental values; and that 
leadership is necessary to guide science. Science 
indicators are seen &8 "social products that must be 
interpreted. .. in a social context." 

Moravcsik, Michael J., "The context of creative science," 
Interseiencia . 1 (July-August 1976): 71-78. File "Measuring 
Output . " 

Discusses quantitative (primarily bibliometric) and 
qualitative (peer review) approaches to evaluating 
scientific research in light of intrinsic and extrinsic 
"goals" of science. 

Hosteller, Frederick, "Selection of papers by quality of design, 
analysis, and reporting," in K.S. Warren, ed., selectivity in 
Information Svstexris ; 98-116. (See below. Sec. II.) 

A brief revelw of largely qualitative criteria of quality 
associated with the items listed in the title. 

Nalimov, V. V. , Measurement of Science; Study of the Development 
of Science as an Information Process , translation from the 
Russian on microfilm (Washington, D.C.: Foreign Technology 
Division, U.S. Air Force Systems Command, Oct. 13, 1971). 

"Measurement of Science" is alternatively translated as 
"scientoaetrics," and Nalimov is credited with inver.t the 
term. The compiler of this bibliography has not Sr 't* ;,!9 
docxament. However, the author says of his book thac ne of 
the chapters in the book was devoted entirely to the 
[Science] Citation Index. In other chapters, the Citation 
Index was used as the main tool for delineating the 
informational aspects of the evolution of science" (V.V. 
Nalimov, Foreword to E. Garfield, Essays of an Information 
Scientist . Vol. 6; see Sec. I. A.). See also E. Garfield, 
"in tribute to V.V. Nalimov: Renaissance scholar and 
scientometrician par excellence," Essays of an Info rmation 
Scientist . Vol. 5 (Philadelphia: ISI Press, 1983): 417-427. 

National Academy of Sciences, The Ouallty of Research i n Science: 
Methods for Postperformance Evaluation in the National Science 
Foundation , report to the NSF by the Stibcommittee on 
Postperformance Evaluation of Research, Committee on Science, 
Engineering, and Public Policy, NAS (Washington, D.C.: NAS Press, 
1982) . Shelf. ABT. 

Discusses general probleifts associated with evaluating basic 
research, some specific techniques of evaluation (peer 
review, bibliometric, case studies of scientific 
discoveries, retrospective analysis) , and a study of the 
research evaluation which is conducted in the Chemistry 
Division of the NSF. Conclusions from the study of the 
CJiemistry Division are: "[1] Postperformance evaluation 
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already occurs at NSF at the project level and at the 
program level, at least within the. . .Division. [2] NSF 
fails to make clear to Congress the degree to which its 
operating procedures incorporate evaluations of past 
performance. [3] No additional methods of postperformance 
evaluation that we know of will significantly improve the 
selection of individual projects. [4] Additional or 
improved strategies of postperformance evaluation should 
concentrate on aggregate — for exeunple, program — levels and 
on such issues as the allocation of resources aunong the 
subfields of a discipline and the support of young 
researchers . " 

Organization for Economic Cooperation and Development, The 
measurement of Scientific and Techni cal Activities; Proposed 
Standard Practice for Surveys of Researc h and Experimental 
Development ("Frascati Manual 1980"), 4th ed. (Paris: OECD, 
1981). Shelf. 

An elaborate and comprehensive guide to the compilation and 
evaluation of data on R&D, intended for the use of 
individuals within national governments who supply such data 
to the OECD. 

Ramachandran, G.N. , "Quality in science," Current Science, 49 
(Feb. 1980): 87-88. File "Measuring Output." 

Brief overview of quality of science, particularly in India. 
Suggests that "research is of high quality if it requires 
considerable thinking, planning, design, and originality, on 
the part of the investigator, and if it has achieved the 
purpose for which it was started." Advocates more applied 
research in India. 

Reingold, Nathan, review of Elkana, et al., eds.. Toward a Metrj^ 
of Science . Isls . 70 (1979): 443-445. File "Measuring Output." 

Suggests that two models of science are implicitly discussed 
in Toward a Metric of Science (Sec. i) . Put 
simplistic ally, one has science as part of society, the 
other has science as separate from, and more universal than, 
culture and society. For various reasons, Reingold likes 
Helton's and Kruskal's papers the best, though he has 
criticisms of each. 

Robinson, Marshall (President of the Russell Sage Foundation) , 
"Toward a better balance?" Reporting from the Russell Sage 
Foundation . No. 6 (May 1985). File "Measuring Output." 

Essay on the relative merits of quantitative and qualitative 
methods in the social sciences, and how to make the latter 
more "reliable." 

Satariano, William A., "Journal use in sociology: Citation 
analysis versus readership patterns," J.ibrary Quarterly, 48 
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(1978): 292-300. File "Heasuring Output. " 

Author's abstract- » fli.lr> f :,;v3r examines the hypothesis that 
citation pattemo pdmllcl readership patterns in 
sociology." He finds that "[Cjitation patterns reflect a 
cross-disciplinary focus that Is not found in the journals 
most of ten read, iljrthec, notation studies underestimate 
the usefulness of popular social science periodicals and 
sptoialty and regional journals in sociology." 

Sherwin, C.W. and Isenson, R.S., First Interim Report on Project 
Hindsight (Washington, D.C. : Office of the Director of Defense 
Research and Engineering, Oct. 13, 1966 [available from NTIS]). 

Traces the antecedents of important weapons. Finds that 
applied research is very important in the "discovery" of new 
weapons. Cf. Comroe and Dripps, and Illinois Institute of 
Technology (TRACES) in Sec. II. . 

Small, H.G. and Griffith, B.C., "The structure of scientific 
literatures I: Identifying and graphing specialties," Science 
Studies . 4 (1974): 17-40. MIT Resource Sharing Center. (Not 
held in files; Part II under Griffith.) 

Smith, Richard and Fiedler, Fred E., "The measurement of 
scholarly work: A critical review of the literature," Educational 
Record , 52 (Summer 1971). File "Measuring Output." 

Focuses on the strengths and weaknesses of a survey of 
faculty conducted by the American Council of Education in 
1964 to determine the quality of academic departments. 
States that "Most ratings of individual faculty members are 
based on publications," and then goes on to elaborate. 

Social Studies of Science , special "Responses and Replies" 
section containing critiques of Martin and Irvine's work on the 
evaluation of "big" basic science, 15 (1985): 525-575. File 
"Martin/Irvine. " 

Contains articles by Krige and Pestre, Mofcd and van Raan, 
Bud, Collins, and a reply by Irvine and Martir. Valuable 
both as a summary of M&I's work and as a comprv- . snsive 
analysis of that work's 3,imitations . 

Staats, Elmer B. , "The General Accounting Office: Appraising 
science and technology programs in the United States," 
Interdisciplinary Science Reviews . 3, Ko. 1 (1978): 7-19. File 
"Measuring Output." 

Discusses role of GAO^ including case studies of assessment 
of S&T. 

Thackray, Arnold, "Measurement in the historiography of science," 
in Elkana, et al., eds.. Toward a Metric of Science (N.Y. : Wiley, 
1978): 11-30. Shelf. 



34 



Investigates the value of quantitative methods for the 
history of science and for contemporary attempts to measure 
science . 

U.S. Congress, House Committee on Science and Technology, Task 
Force on Science Policy, "An agenda for a study of government 
science policy," Dec. 1984. Pile "Measuring Output." 

An outline of the first major review of science policy in 
about a decade. Includes discussions of peer review and 
other approaches to measuring quality in science. See also 
paper by La Follette tinder Ganz (Sec. IV) . 

U.S. Congress, Office of Technolocfy Assessment, The Health of the 
scientific and Technical Enterprise (Washington, D.c.i OTA, 
1979). File "Measuring Output" and Shelf. 

Includes short section entitled "Diagnosis of Health," which 
asks questions for further research for the most part. 

Warren, Kenneth S., ed., selectivity in inf ormation Systems: 
Survival of the Fittest (N.Y.i Praeger, 1985). Shelf. 

Focuses on bibliographic retrieval and is motivated by a 
concern about the burgeoning medical literature. Includes 
papers by the Coles, J.T. Bruer, and Hosteller. The Coles' 
article is essentially a reworking and summary of their NSF 
study. (See Sec. V.) Bruer 's paper is an attempt to develop 
indicators which can be used with literature retrieval 
systems. (See Sec. II.) Hosteller focuses on standards of 
criteria which physicians might use in assigning priorities 
to the papers which they should read. (See Sec. II.) 
Patterson and Bailar have a very useful literature review of 
research on peer review. (See Sec. V.) See also Warren 
(Sec. II.), Goffman (Sec. I. A), and Stam (Sec. VI.). Three 
other papers by Larkin, Small, and Horowitz, et al., focus 
on various aspects of information processing and are not 
listed separately in this bibliography. 

Warren, Kenneth S., "Selectivity within the ecosystem of 
scientific communication," in K.S. Warren, ed.. Selectivity in 
^information Systems ; 3-9. (See Sec II.) 

Discusses the retrieval of citations from computerized 
information systems. Advocates "egalitarian" input of 
cites, but "elitist" output. That is, everything is put in, 
but only cites from select journals are put out until 
resources allow for more thorough service. 

Zumeta, William, "Anatomy of the boom in postdoctoral 
appointments during the 1970s: Troubling implications for quality 
science?" Science. Technology & Human Values . 2, No. 2 (Spring 
1984): 23-37. Attached is Lee Grodzins, "Commentary on Zumeta's 
article," science. Technoloefv & Human Values . (Spring 1984): 
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38-39 • File "Education." 

Uses data from three national surveys to address the 
following questions: "What accounts for the rapid growth in 
the incidence of postdoctoral activity since the late 1960s? 
2. Who are the 'New Postdoctorals • • • .and how are they 
different from earlier postdoctorals in terms of 
motivations, key demographic characteristics (i.e., sex) 
and, most importantly, quality? 3. To what ext^T^nt has the 
location of postdoctoral activity shifted, as its level has 
increased?... 4. What can be said about changes in the 
quality and character of postdoctoral training as its 
incidence has increased dramatically?" (Emphasis added.) 

Ziman, John, "From parameters to portents — and back," in Elkana, 
et al., eds.. Toward a Metric of Science (N.Y.: Wiley, 1978): 
268-2f)3. Shelf. 

Examines the potential for and problems with quantifying 
outputs of science. States that "clearly knowledge [as an 
output of science] is not an economic category that can be 
quantified. This. . .statement. . .rests upon simple 
fundamental principles. Any scheme for the quantification 
of knowledge would constitute a theoretical representation 
of certain processes in the real world. Within such a 
scheme, the logical relations between symbols standing for 
real operations like the creation of knowledge and its 
communication/ aggregation and use, must be, of course, 
essentially equivalent to relations between the real 
processes they represent. Thus, the abstract algebra of the 
symbolic system. . .must be isomorphous with the intrinsic 
relational structure of the reality it is supposed to 
depict. A model that violates this principle can produce 
nothing but nonsense. 

"Whatever it may be, the intrinsic logic of knowledge does 
not constitute an algebra with typical arithmetic 
properties. . . " 

Compare Abt Associates report, A Methodologica l Approach, >> 
(Sec. II) and paper by Ganz and Sneed (Sec. VI). 



III. Works Dealing Specif icallv with "Science Indicatorc- 

Brooks, Harvey, "Science indicators and science policy 
sclentometrics , No. 5-6 (1980); 331-337. File "Science 
Indicators . " 

Discusses the definition of the research system, society's 
expectations of science, and the strengths and weaknesses of 
quantitative indicators, such as those used in Science 
Indicators . 
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Brooks, Harvey, "Science indicators and science priorities," in 
La Follette,. ed. , Quality in Science (Cambridge, Mass.: MIT 
Press, 1982): 1-32. Shelf. 

Views science indicators as a means to evaluate the progress 
of science and as a tool to set research priorities. 
Disc\i5ses several sets of problems with prevalent science 
indicat-crs: setting the appropriate balamce between factors 
intriKsic and extrinsic to science in specifying input and 
output indicators; focusing on international as well as 
national scientific activities; and, more broadly, relating 
research priorities to social and economic needs. 

Cole, Stephen, et al., "Measuring the cognitive state of 
scientific disciplines," in Elkana, et al., eds.. Toward a Metric 
of Science (N.Y.: Wiley, 1978): 209-251. Shelf. 

From the. authors' introduction: "Problems occur in 
interpreting the indicators presented in Sl-72 . . .Two fields 
appearing to be similar on a set of quantitative indicators 
may be qualitatively in quite different stages of 
development and exhibit very different cognitive 
structures. . .Clearly we need Indicators of the cpialitative 
as well as the quantitative aspects of science." However, 
the Indicators of qualitative aspects of science used 
in the paper are themselves quantitative (publications 
and citations) . 

Duncan. Otis Dudley, "Science indicators and social indicators," 
in Elkana, et al., eds.. Toward a Metric of Science (N.Y.: John 
Wiley & Sons, 1978): 31-38. Shelf. 

Develops a critique of the two sets of indicators based on 
the view that "Science and society are of a piece" and that 
in evaluating their health their close interaction should be 
considered. (See also book by Duncan listed in Sec. 
VIII. A.) 

Elkana, Yehuda, "Images of knowledge, qualitative indicators and 
science policy," in Proceedings of th e First International 
conference on Sociaq Studies o f Science. Nov. 4-6, 1976 (Society 
for Social Studies of Science, 1976). File "Science Indicators." 

"The point of this paper is that the desirable science- 
policy should emerge out of a critical dialogue on competing 
images of knowledge. The correct science-policy is neither 
independent of the body of knowledge, nor does it follow 
unambiguously from the body of knowledge. It depends on 
images of knowledge serving as indicators." 

Evenson, R., "Technology indicators: Comments on SITS," "p^^er 
presented at the third review symposium on NSB Science 
Indicators, May 16, 1980, Washington, D.C. File "Science 
Indicators." 
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Evenson develops an econometric approach to the measurement 
of research productivity. 

Ezrahi^ Yaron^ "Political context of science indicators^" in 
Elkana, et al., eds., Toward a Metric of Science (N.Y.: Wiley, 
1978): 285-327. ShelJ- 

A study of the interaction between values , politics, and 
measurement with respect to science indicators. 

Garfield, Eugene, et al., "Citation data as science indicators," 
in Elkana, et al., eds.. Toward a Metric of Science (N.Y.: Wiley, 
1978): 179-207. Shelf. 

Includes discussions of co-citation and bibliographic 
coupling, the social and cognitive structure of science, and 
models of scientific research. 

Greenberg, Daniel S., "NSF finds science in the US alive and 
well," Science and Government Report , £, No. 20 (Dec. 1, 1979). 
File "Measuring Output. 

Journalistic summary and review of Science Indicators 1978 . 

Greenberg, Daniel S., "On the state of American research," 
Washington Report (Dec. 27, 1979): 1455-1456.^ File "Measuring 
Output." 

Critical review of SI 78 , summarizing arguments by GAO. 
(See U.S. Congress, GAO, Sec. III.) 

Holmfield, John D^, "Science indicators and other indicators: 
Some user observations," 4S (Society for Social Studies of 
science^ Newsletter , 3 (Fall 1978): 36-43. File "Science 
Indicators. " 

The author is on the staff of the House Committee on Science 
and Technology. "In this paper the focus will be on the 
lessons and insights that have been learned from the 
development and use of indicators in a number of other 
fields and their applicability to SI." Discusses various 
sorts of indicators (e.g. of water quality, housing 
quality) . 

Kitti, Carole, notes summarizing meeting on "Technology 
indicators: Recent developments in the use of patent data," 
Technology and Culture , 24 (July 1983): 473-475. File "Science 
Indicators." 

Kochen, Manfred, "Models of scientific output," in Elkana, et 
al., eds.. Toward a Metric of Science (N.Y.: Wiley, 1978): 
97-136. Shelf. 

The paper attempts "To clarify thre*^ conceptual issues 
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central to evaluating. . .information systems [to help in 
planning science] ; To compare five models for S&T and S&T 
planning; To evaluate various output measures for the five 
models; To suggest priorities for future research in this 
area and for constructing new science indicators." 

Kruskal, William, "Taking data seriously," in Elkana, et al., 
eds.. Toward a Metric of Science (N.Y. : Wiley, 1978): 139-169. 
Shelf. 

Argues that "documentation of... [data going into sl-721 's 
modes of genesis is weak"; that "there is remarkably 
little study or consideration [in SI-72 1 of the limitations 
of the data used: arbitrariness or vagueness of definition, 
degree of random error components, errors of bias, and so 
on"; and "Third, there are some more technical statistical 
problems . " 

La Porte, Todd R. and Chisolm, Donald, "Indicators of the 
public's attitudes toward science and technology: Science 
indicators : 1972. 197A and 1976 . a review and prospective 
reflections," paper prepared for Symposium on Science Indicators, 
Social Indicators Group, Social Science Research Council, May 
1978. File "Science Indicators." 

Critiques existing survey designs and proposes new ones. 

MacAulay, James, "The ghost in the big machine: Science 
Indicate rs/ 19 7 6, " 4S fSocietv for Social Studies of Science) 
Newsletter . 1 (Fall 1978): 30-35. File "Science Indicators." 

Critique of SI 76 . 

Mcculloch, Rachel, "International indicators of science and 
technology: How does the U.S. compasre," paper presented at the 
SSRC Review Symposium on SI-76 (May 1978). File "Science 
Indicators . " 

Criticizes SI-76 on the grounds that the relationships 
between inputs to and outputs from scientific resp-rch are 
not well enough understood to make international aparisons 
which have sufficient validity to guide policymak:-„s. 

McGinnis, Robert, "Science Indicators/1976: h critique," 4S 
f Society for Social Studies of Science) New sletter. 1 (Fall 
1978): 14-29. File "Science Indicators." 

Review of SI 76 in light of Elkana, et al., eds.. Toward a 
Metric of Science . (See Sec. i.) 

Mazlish, Bruce, "The quality i^>f 'The quality of science': An 
evaluation," in La Follette, ed. , >ualitv i n Science (Cambridge, 
Mass.: MIT Press, 1982): 48-67. Shelf. 

Suggests that science indicators are inherently value-laden; 
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that quantitative indicators deal primarily with factors 
intrinsic to science and "can block other modes of 
understanding"; "that historical analogies. . .and case 
studies are alternate modes of understanding the quality of 
science..."; that science is, in part, a political process 
involving choices about fimdsunental values; and that 
leadership is necessary to guide science. Science 
indicators are seen as "social products that must be 
interpreted. .. in a social context." 

Morison, Robert S., "Needs, leads, and indicators," in La 
Follette, ed.. Quality in Science (Cambridge, Mass.: MIT Press, 
1982): 33-47. Shelf. 

Argues that it is easier and more fruitful to devise 
indicators for applied than for basic research, as it is 
clear "that indicators should tell us something about how 
science is doing in relation to what it is supposed to do 
for h\aman welfare." 

"NSF finds science in the US alive and well," Science and 
Government Report . 2. (Dec. 1, 1979): 1-5. File "Science 
Indicators. ^ 

Journalistic account of SI-78 . 

National Science Foundation, Characteristics of Recent Science/ 
Engineering Graduates; 1982 (Washington, D.C.: NSF, 1984). 
Shelf. 

National Science Foundation, Manpower Resources for Sc ientific 
Activities at Universities and Colleges. January 1976 
(Washington, D.C.: NSF, 1976). Shelf. 

National Science Foundation, National Patterns of RSD R esources: 
Funds and Manpower in the United States (Washington, D.C.: NSF, 
1977). Shelf. 

National Science Foundation, National Patterns of Sci ence and 
Technology Resources: 1984 (Washington, D.C.: NSF, 1984). Shelf. 

Sections on R&D performance, R&D/GNP ration, relative funding 
of basic and applied research, international comparisons, 
and S&T personnel; statistical tzibles. 

National Science Foundation, Science and Engineering Personnel: A 
National Overview (Washington. D.C.: NSF, 1985K Shelf. 

See especially Chap. II, "labor market indicators," pp. 
17-22. 

National Science Board,, Science Indicators (Washington, D.C.: 
National Science Foundation, 1972, 1976, 1978, 1980, 1982, 1985). 
Shelf. 
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Organization for Economic Cooperation and Development, dociaments 
related to a Conference on Science and Technology Indicators held 
in Paris, Sept. 15-19, 1980. File "Science Indicators." 

Annoxincement of Conference; List of Papers and Schedule of 
"Workshop IV"; papers by Serim, Chapman and Gibbons, 
Johnston, Wright, Pullman; and Preliminary Report on 
Conference. 

Price, Derek de Solla, Toward a model for science indicators," in 
Elkana, et al., eds. , Toward a Metric of Science (N.Y.: Wiley, 
1978): 69-95. Shelf. 

Praises NSF's Science Indicators as a con^ilation of data 
useful to those studying science and science policy. 
Advocates that future attempts to develop science indicators 
focus on basic and applied research, rather than 
development, where researchers are defined operationally as 
those who produce published papers. Suggests procedures for 
evaluating the economic impact of "researchers." 

U.S. Congress, General Accounting Office, Office of the 
Comptroller General, "Science indicators: Improvements needed in 
design, construction, and interpretation" (Washington, 
D.C.: GAG, PAD-79-35, Sept. 25, 1979). File "Science 
Indicators . " 

A lengthy and detailed critique, primarily of SI 76. From 
the abstract: "They [the NSB and NSF] should emphasize a 
more conceptual ataproach which first identifies what will be 
measured, and then generates the appropriate data. Attempts 
should be made to devalop indicators of the process and 
substance of research «ind to better differentiate betweeen 
science and technology. More interpretation of the meaning 
of indicators should be included in future reports." 

Zeisal, Hans, "Difficulties in indicator construction: Notes and 
queries," in Elkana, et al., eds.. Toward a Metrjc of Science 
(N.Y.: Wiley, 1978): 253-258. Shelf. 

Notes on basic problems with Science Indicators and the 
relationship between indicators and indices. 

Zuckerman, Harriet and Miller, Roberta Balstad, eds., "Science 
indicators: Implications for research and policy," special issue 
of Seientometrics . 2 (Sept., Nov. 1980 [Nos. 5-6]): 327-448. MIT 
Dewey Library. 

Based on a conference sponsored by the Social Science 
Research Council held in May 1978. Papers by H. Brooks (see 
this section), H. Averch, H. Zuckerman & R.B. Miller, R. 
Mcculloch, R.B. Freeman, C.V. Kuh, J. Ben-David, R. Bowers, 
H,,V. Riecken, and T.R. LaPorte, and shorter comments by 
other authors. 
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IV. Forecasting and Research Priorities 



Abt Associates, "A comparative study of the prospective and 
retrospective approaches to the evaluation of proposed basic 
research," report to the National Science Foundation (Cambridge, 
Hass.: Abt, July 1969). Shelf. AB'r. 

Investigates "the possibili^.y of developing measures that 
could be used In an operati.onally realistic way to 
anticipate future effectiveness of present investment 
alternatives in basic research." "...measures are based on 
changes in basic knowledge proposed by researchers seeking 
financial support, and the use made by siabsequent 
investigators of laiowledge gained from completed projects." 
An ambitious and probably unique attempt to precisely 
specify potential and realized advances in knowledge. 

Ahrenp, H.J., et al., "Priorities in research policy," Research 
Policy; , 2. (1973): 94-126. File "Priorities and Foresight." 

One hundred German research planners and managers were asked 
to "evaluate 13 research areas sponsored by the government 
[education, defence, health,...] according to 8 objectives 
[raising of the technological standard of industry, 
improvement of international cooperation,...]" 

Brooks, Harvey, "ModelR for science planning," Public 
Administration Review , H (May/June 1971); 364-374. File 
"Priorities and Fore^^it^ht. " 

The paper discut; ?s "four models of the research enterprise 
which might serv^e as partial bases or rationales for 
planning the support and development of science. . .Each 
implies different mechanisms and criteria of choice, and the 
scientific system as a whole should consist of a mixture or 
blend of different allocation mechanisms in which the 
criteria appropriate to one of the four models are 
dominant." The four rationales are cultural, economic, 
social and educational. The four corresponding models are: 
"science as a quasi-autonomous. . .activity" ; "science as a 
technical overhead"; "science as a social overhead"; and 
"science as a tertiary industry or consumption good." The 
second and fourth models are elaborated by Weinberg and 
Toulmin, respectively, in papers listed below in this 
section. 

Brooks, Harvey, "The problem of research priorities," Daedalus , 
107 , No. 2 ("Limits of Scientific Inquiry"; Spring 1978): 
171-190. Shelf. 

Discusses the interrelationship between scientific and 
social priorities, supply of scientific talent vs. supply of 
funds as limits to scientific activity, and "proposal 
pressure" as a means to evaluate research priorities. The 
notion that peer review "works" with respect to the 
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assessment of "truth," but not utility, Is widely cited in 
papers on the subject of peer review. 

Brooks, Haivey, "Science indicators and science policy," 
Scientometrlcs , Z, No. 5-6 (1980): 331-337. File "Science 
Indicators." 

Discusses the definition of the research system, society* s 
expectations of science, and the strengths and weaknesses of 
quantitative indicators, such as those used in Science 
Indicators . 

Brusilovsky, B. Ya., "Partial and system forecasts in 
scientometrics," Technological Forecasting and Social Change , 12 
(August 1978): 193-200. File "Priorities and Foresight." 

Author's abstract: "Science itself can be considered as a 
* fuzzy system.* In attempting to deal with possible laws of 
scientific development we formulate a simple, partial model 
and illustrate its use as a means to control the strategy of 
investments in science." Systems analysis approach. 

Carter, Sir Charles, "Conditions for the successful use of 
science," Science (1982). 

"In some areas of intense scientific effort (in Britain) , 
commercial priorities t^nd to be forgotten. . .Evidence 
suggests that the successful use of science depends on the 
overall quality of management rather than specifically on 
its degree of scientific knowledge." 

Carter, Luther J., "More stress on applied science at NSF," 
Science , 205 (August 17, 1979): 675-676. File "Priorities and 
Foresight . " 

Journalistic account of changes under the Carter 
Adminlstrat ion . 

Chynoweth, A.G., "Science-engineering coupling and some 
priorities in materials research," Annual Review of Materials 
Science , ^ (1975): 27-42. File "Priorities and Foresight." 

Suggests that social needs should guide priority-setting in 
materials science, and discusses the Implications of this 
proposition in some detail. 

Coward, H. Roberts and Franklin, J. Jeffrey, "Setting priorities 
for science-driven technology: Patents and bibliometric models," 
in Carole Ganz, ed.. The Rise and Fall of Priority Fields , U.S. 
papers presented at a National Science Foundation/Exiropean 
Science Research Councils Symposium, Paris, France, Sept. 22-24, 
1985. Shelf. 

Investigates the relationship between basic research and 
technological innovation using co-citation maps to represent 
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fields within the foinner and patent data to represent the 
lattar. Focuses on solid state physics and concludes that 
**the strong pattern of interaction between the biblionetric 
model emd patenting activity ^n solid state devices suggests 
that the technique night be used in order to identify 
research-driven technologies and technology^drivlng research 
areas . . • [This] is increasingly important for research 
priority setting.'' 

Ganz, Carole y ed. , The Rise and Fall of Priority Fields ^ U.S. 
papers presented at a national Science Foundation/European 
Science Research Councils Symposium, Paris, France, Sept. 22-24, 
1985. Shelf. 

Bode Bartocha, head of NSF*s Division of International 
Programs, gives a historical overview; Coward and Franklin 
of the Center for Research Planning investigate the 
integration of patent and bibliometric statistics to better 
understand the science-technology interface (see immediately 
eibove) ; Ganz and Sneed offer an application of concepts from 
the philosophy of science to problems of setting priorities, 
with a valuable bibliography (see Sec. VI); La Follette 
discusses the ongoing study of U.S. science policy by the 
Hom^e Committee on Science snd Technology. There are also 
papers by Ratchford, May field, Ryer, and Willenbrock. 

Garfield, Eugene, "The economic impact of res&arch and 
development," Current Contents (Dec. 21, 1981): 5-15. File 
"Priorities and Forecasting." 

Includes data on the economic return of basic research and 
summaries of Edwin Mansfield's work. Has a useful 
bibliography. 

Hoaglin, David C. , et al.. Data for Decisions (Cambridge, Mass.: 
Abt Associates, 1982) . Shelf. 

Investigation of methods for gathering data to be used in 
formulating public policy. Includes a "quality checklist" 
for each method. 

Irvine, John and Martin, Ben R. , Foresicfht in Sciencej Picking 
the Winners (London: Frances Pinter, 1984). Shelf. ABT. 

Study of research forecasting. Includes cross-national 
comparisons (U.K., France, West Germany, U.S., Japan). U.S. 
section covers COSPUP field surveys and NSF commissioned 
studies. Addresses both basic and "strategic'* research. 

Krauch, Helmut, "Priorities for research and technological 
development." Research Policy . 1 (1971/72): 28-39. File 
"Priorities and Foresight." 

Results of a piiblic opinion poll on rasearch priorities in 
West Germany are compared with government expenditures. The 
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priorities, determined from the two sources, diverge 
significantly. It is concluded that interest groups have 
too much influence. 

Lederman, Leonard L. , "Foresight activities in the U.S.A.: Time 
for a re-assessment?" Lona Rang e Planning, il. No. 3 (1984): 
41-50. File "Priorities and Foresight." 

Does not focus on forecasting the results of scientific 
research, but is nonetheless of some value as a generic 
summary, especially because the author was with the NSF when 
he wrote the piece. 

Leeper, Pepper, "Which disease? What vaccines? How to set 
priorities," ^' cws Report (of the National Academy of Sciences), 
35 (Dec. 1984-Jan. 1985): 4-11. File "Disciplinary." 

Discusses priority- setting in research on public health. 

Lepkowski, Will, "Public concern: New force in science policy," 
Chemical S Engineering News (March 29, 1982): 43-44. File 
"Priorities and Foresight." 

"Social context and public disenchantment with science and 
technology must be considered in making science policy, 
recent reports say." Discusses OECD report by H. Brooks and 
contrasts it favorably with statements by George Keyworth. 

Libraary of Congress, Congressional Research Service, Science 
Policy Division, "An overview of the decision making process in 
the areas of research and development priorities," prepared for 
the Office of Technology Assessment Technology Assessment 
Advisory Council, May 26, 1976. File "Priorities and Foresight." 

Includes sections entitled "The peer review system" and 
"Methodology for evaluating the economic and social impacts 
of R and D expenditures." 

Maddox, John, "Choice and the scientific community," in Shils, 
ed., criteria for Scientific Development (Cambridge, Mass.: HIT 
Press, 1968): 44-62. Shelf. 

Contrasts choice of research topic and of the applications of 
findings. Concludes that "the wisdom of the choices which 
need to be made both in academic science and in its 
application will be determined by the intellectual cohesion 
of the scientific community as such." 

Martin, Ben R. and Irvine, John, "Internal criteria for 
scientific choice: An evaluation of research i^^^igj-fn^^^y . 
physics using electron accelerators," MAn^rvg. 19 (Autumn 1981) : 
408-432. File "Martin/Irvine." See also entry under Irvine. 

A discursive analysis of the need for more systematic 
methods of research evaluation at CERN and other facilities. 
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PuJolic Agenda Foundation, Science Policy Priorities and the 
PuiDiloi A Ppp97rt 9^^ a Pilot ProSect to Assess Public Attitudes 
About PriorJ^tles mnd Tndic ators of Ouallty_^or^Sclenti£ic 
Research (Prepared for the project "Assessment of science: 
Development and testing of indicators of quality," Harvard 
University, 3.980) • Shelf. 

Authors* preface: ^'This repoxi: summarizes the findings of a 
pilot study on public attitudes 1:ovard a range of 
controversial issues relating, in general, to science and 
technology 2uid quality of life. Specifically, the Public 
Agenda assembled small groups of citizens to probe their 
reactions to a variety of scientific research proposals. The 
goal of the pilot study, as described in the proposal, was 
to generate hypotheses about implicit and explicit criteria 
people use to assess the impact of science and technology on 
the quality of life. 

**The project was designed to explore the following 
questions: 1. Can the public make a contribution to the 
development of quality indicators? ... 2 . VThat criteria do 
average citizens use to assesF research piorities? . . .3 . Do 
average citizens approach scie2iCe as scientists do?... 4. Are 
there measurable differences be'" ^n superficial public 
opinion and considered public j% vj .i^t?" 

A number of hypotheses are then proposed, based on the 
research . 

Ross, H.H., et al., "Setting research priorities," Research 
Policy , 8 (1979): 260-272. File "Priorities and Foresight." 

Authors* abstract: "An algorithm, specially developed to set 
research priorities, was used. .to rank ten different 
projects. . .previously ranked. . .by personal judgement. 
Comparison of the results from both methods indicates 
excellent agreement among rankers within each method, little 
discrepancy between the two methods in final rankings, and 
no apparent advantage to the use of the algorithm." 

Snellen, I. Th. K. , "Social merit as a criterion of scientific 
choice: Its application in Dutch science policy," Minerva , 21 
(Spring 1983): 16-36. File "Priorities and Foresight." 

Uses a method for assigning weights to social goals 
involving "the comparison of pairs of alternative 
programmes." Acknowledges Steinberg's early work in the 
field (see below, this section). Includes a very good 
discussion of science policy-making in the Netherlands. 

Thomas, Lewis, "On the planning of science," in H.H. Fudenberg 
and V.L. Melnick, eds.. Biomedical Scientists and Public Policy 
(N.Y.: Plenum Press): 67-76. Widener R.854 U5 B53. 
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Toulnln, Stephen, "The complexity of scientific choice: A 
stocktaking," in Shils, ed., Criteria for Scientific DeveloTament 
(Cambridge, Mass.: MIT Press, 1968): 63-79. Shelf. 

Reviews and extends the arguments presented by Polanyi, 
Weinberg, Carter, and Maddox on the svibject of scientific 
choice. (All of these articles are repriiited in the MIT 
Press volume; Weinberg's and Maddox' are listed in this 
section of the current bibliography.) Suggests in 
conclusion that the research contract system, whereby 
private institutes such as Rand conduct the government's 
research, successfully combine scientific expertise, 
administrative flexibility, and financial responsibility, 
and should be tried in Britain (from where Toulmin writes) . 

Toulmin, Stephen, "The complexity of scientific choice 11: 
Culture, overheads or tertiary industry?" in Shils, ed.. Criteria 
for Scientific DeveloTsment (Cambridge, Mass.: MIT Press, 1986): 
119-133. Shelf. 

Discusses two justifications for funding basic research: 
basic research as an overhead (see Weinberg below, this 
section) and basic research as high culture. Proposes as an 
alternative the concept of basic research as a tertiary 
industry, which will maintain employment after primairy and 
secondary industries are in decline. Also discusses 
problems with quantification and evaluation of the economic 
(employment) effects of research. 

United Nations Educational, Scientific and Cultural Organization, 
Method for Priority Determination in Science and Technology: 
UNESCO/UNACAST Surveys of Institutional Needs of developing 
Countries in the field of Science and Technology (Paris: UNESCO, 
1978). Shelf. 

"The disclosure of [areas of priority] is effected through 
an assessment of the relevance of scientific emd 
technological disciplines for national development 
objectives and programmes. The assessment is made by panels 
of experts drawn from governmental departments and from 
scientific institutions. . .The output of the method is a 
series of priority charts and profiles showing the relative 
merits of fields of science and technology for the 
achievement of development objectives, and the relative 
dependence of the latter on the former." 

United Nations Educational, Scientific and Cultural Organization, 
"Seminar on evaluation of research priority determination methods 
in science and technology (Paris, 27-30 September 1983): Final 
report," (SC-83/CONF. 731/3, Sept. 30, 1983). File "Priorities 
and Foresight." 

summarizes the results of a planning exercise 
using the method. (See immediately above.) 
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United Nations Educational, Scientific and Cultural Organization, 
The UNESCO Method for Prior ity Determination in Science and 
TechnolocTV (Paris: UNESCO, 1981??). Excerpts on Shelf. 

National case studies of the application of the UNESCO 
method for priority determination: Argentina, 
Australia, Costa Rlca^ Jordan, Nepal, Peru, Portugal. (See 
above, under UNESCO in this section, for description of the 
method . ) 

U.S. Congress, Committee on Science and Astronautics, report by 
the National Academy of Sciences, Basic Res earch and National 
Goals (Washington, D.C.: USGPO, March 1965). File "Priorities 
and Foresight." 

The report includes papers by several scientists and science 
policy experts. Only the summary of the report is in the 
file. 

Warsh, David, "New indax shows high-technology jobs on the 
decline," Boston Globe (August 7, 1979). File "Priorities and 
Foresight. " 

Disucsses forecast by Dere?c de Sella Price using a 
"high-technology recruitment index," based on help-wanted 
advertising in 40 periodicals and newspapers. 

Weinberg, Alvin M. , "Criteria for scientific choice," in Shils, 
ed., criteria for Scientific Development (Cambridge, Mass.: MIT 
Press, 1968): 21-33. Shelf. 

Distinguishes "internal" from "external" criteria. The 
former are 1/ "Is the field ready for exploitation?" and 2/ 
"Are the scientists in the field really competent?" 
Weinberg observes that only experts active in the field can 
answer these questions. External criteria are technological 
merit, scientific merit (i.e. whether research in one field 
fertilizes other fields), and social merit. The author 
concludes "that the most valid criteria for assesing 
scientific fields [which are to be publicly funded] come 
from without rather than from within the scientific 
discipline that is being rated." 

Weinberg, Alvin, "Criteria for scientific choice II: The two 
cultures," in Shils, ed. , Criteria for Scien tific Development 
(Cambridge, Mass.: MIT Press, 1968): 80-91. Shelf. 

Develops the idea of funding for basic research as em 
overhead in relation to the whole scientific and 
technological enterprise* The abso3.ute amount of funding is 
determined accordingly, through the political process. 

Weinberg, Alvin M. , "Values in science: Unity as a criterion of 
scientific choice," Minerva , 22 (Spring 1984): 1-12. File 
"Priorities and Foresight." 



53 



48 



"Truth" is posited as the underlying value of scientific 
practice, "unity" as that of scientific adminsistration, 
vhere "administration" means "the art of choosing, among 
infinitely many possible questions answerable by science, 
vhich questions to ask." The criterion of "unity" "stems 
directly from a belief that a unified body of scientific 
taiowledge, in which the different parts of science are 
related to each other, are more consistent with each other, 
and illuminate each other, is in a veiry fundamental sense 
better~a greater intellectual achievemnt, more powerful, 
more beautiful — than an aggregate of scientific knowledge 
not so unified. " 

Yankelovich, Daniel and Lefkowitz, Bernard, "The public debate on 
growth: Preparing for resolution," Technological Forecasting and 
social Change . 11 (1980): 95-140. File "Priorities and 
Foresight." 

"This report consists of three parts. The First ... presents 
our summary interpretation of a wide body of social research 
data showing the attitudes, beliefs, and values Americans 
currently hold on the subject of economic growth... The 
research reveals a picture of Americans midway 
between. . .unfettered optimism. . .and a growing psychology of 
limits. The second part. . .considers how this ambivalent 
state of mind might resolve itself in the future... we have 
included a tabular presentation of survey data as a 
reference source for analysts . . . 



V. peer Review 

Anderson, Richard C. , et al., "Public. :', a ratings versus peer 
ratings of universities," Journal of the American S ociety fgy 
TnfQrmation Science (March 1978): 91-103. File "Measuring 
Output." 

"Presents a quantitative comparison of peer versus 
bibliometric procedures ... The peer ratings used are the 
Roose-Anderson rating of the quality of graduate faculty in 
10 scientific fields." Bibliometric ratings are based on 
numbers of papers and citations. 

Cole, Jonathan R. and Cole, Stephen, "Chance and consensus in 
peer review," Science . 22A (Nov. 20, 1981): 881-886. File "Peer 
Review." 

Author's summary; "An experiment in which 150 proposals 
submitted to the National Science Foundation were evaluated 
independently by a new set of reviewers indicates that 
gettirg a research grant depends to a significant extent on 
chance. The degree of disagreement within the population of 
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eligible reviewers indicates that whether or not a proposal 
is funded depends in a large proportion of cases upon which 
reviewers happen to be selected for it. No eystematic bias 
in the selection of HSP reviewers was found." Based on the 
Coles' study of the NSF peer-reviev system (see below) . Seta 
also critique by Hamad, listed in this sectiion. 

Cole, Jon;ithan R. and Cole, Stephen, "Experts" »consensns» and 
decision-making at the National Science Foundation," in K.S. 
Warren, Select ivitv in Information Svstems ; 27-63. (Shelf; see 
Sec. II. } • 

Largely a reworking of the Coles' article in Scieiufi, 
"Chance and consensus in peer review." (See immediately 
aODove.) Both articles are based on the Coles larger study of 
peer review in the NSF (immediately below). The conclm^xon 
presents some new views on consensus and lack thereof in tY\o 
process of peer review. 

Cole, Stephen, Rubin, Leonard, and Cole, Jonathan R., Peer Review 
in the National Science Foundation; Phase Orie of a Study ; Cole, 
Cole, and NAS Committee on Science and Public Policy, Peer Review 
in the National Science Foundation; Phase Tvro of a Study 
(Washington, D.C.; National Academy of Sciences, 1978, 1981). 
Phase One on Shelf. 

The authors analyze 1,200 NSF proposals and make five major 
conclusions (quotes from p. 29 of Coles' paper in Warren, 
ed. , listed immediately above); "i. There is a high 
correlation between reviewer ratings and grants made... 2. 
...there was not a high correlation between grants awarded 
and measures of the previous scientific performance of the 
applicants... 3. Reviewers at major institutions did not 
treat proposals from applicants at major institutions more 
favorably than did reviewers from lesser institutions... 4. 
Professional age (length of career) had no strong effect 
either on ratings received or on the probability of 
receiving a grant. 5. There were low or ^s^oderate 
correlations between reviewer ratings (and the funding 
decision) and the following characteristics of the 
applicants; prestige ra^ik of current academic department, 
acadeiuic rank, geographic location, NSF funding history over 
the previous five years, and locus of PhoD. training." 

"Congressmen Conlan and Bauman attack NSF peer review," Physics 
Today, 28 (Sept. 1975); 77-78, 80. File "Peer Review." 

Journalistic account of one of the periodic Congressional 
attacks on the NSF» Congressmen ask. Who reviews the 
(possibly biased) reviewers? 

Gutitafson, Thane, "The controversy over peer review," Science , 
liUO (December 12, 1975); 1060-1066. File "Peer Review." 



Cogently defends peer revie:^ against a number of critics 
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While suggesting some modifications in the system (s) . 

Hamard^ Steven^ ed. , "Peer commentary on peer review: A case 
study in scientific quality control^" The Behavioral and Brain 
sciences , 5^ No. 2 (Cambridge, England: Cambridge University 
Press, 1982). File "Peer Review," 

This document includes about 50 short commentaries on the 
effectiveness of peer review In determining the quality of 
research. The articles focus on a controversial study by 
Peters and Ceci in wuks >i previously rejected papers 
were resubmitted to the same journals under fictitious 
authors' names. Many of these resubmitted papers were 
accepted . 

Hamad ^ Steven, "'Rational disagreement in peer review ," Science > 
Technology , & . Human Values . special issue entitled "Peer review 
and public policy," 10, No. 3 (Summer 1985): 55-62. Shelf. 

Critiques the Coles* paper ^ "Chance and consensus in peer 
review" (see above) , using statistical arguments and th3 
guiding notion that "a certain degree of disagreement is 
not only a healthy but an informative and even essential 
aspect of scientific activity and that it should be clearly 
represented as such, rather than as some capricious and 
arbitrary shortcoming of scientific judgement^ responsible 
only for unfairness and waste" (p. 59) . 

HiltSr Philip J.^ "Low-ranked social science projects moved to 
top," WashinqtoTi Post (October 2, 1983): A2. File "Peer Review." 

A possible case of overt political intrusion into the peer 
review process; illustrates the relative vagueness of the 
criteria by which social science research is assessed. 

Patterson, Kay and Bailar, O'ohn C. Ill, "A review of journal peer 
review," in K.S. Warren, ed.. Selectivity in Information Systems; 
64-82. (See Sec. II.) . 

A valuable review of tr.e literature. 

Ravetz, J.R., Scientific Knowledge and Its Social Problems 
(Oxford: Clarendon Press of Oxford University Press, 1971), Chap. 
10, "Quality control in science." File "Measuring Output." Book 
in Cabot Library Q.175 .R3. 

Argues for qualitative (peer review) approach to evaluating 
scientific research, though does not explicitly employ the 
terms "qualitative" and "quantitative." The author observes 
that "It is impossible to design a simple set of routine 
tests, by which one could assign some numerical marks to a 
scientific problem, and then grade it on a linear scale. Nor 
indeed would it be feasible to erect a formal system of 
categories of quality, and train up a corps of expert 
assessors... to operate in their framework. For the 
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techniques are so subtle, the appropriate criteria of 
adequacy and value so specialized, and the materials so 
rapidly changing, that any fixed and formalized categories 
would be a blunt and obsolete instrument as soon as it were 
brought into use. The result of this special characteristic 
of science is that if there are to be truly expert 
assessments of quality of work, they mist be made by a 
section of those who are actually engaged upon that work." 

Roy, Rustxim, "Peer review" (letter to the editor), Phvsics Today , 
29 (July 1976). File "Peer Review." 

Addresses possible abuses of peer review system investigated 
by the Symington subcommittee in 1975. 

Science^* Technology, & Human Values . 10 , No. 3 (Summer 1985), 
special issue entitled "Peer review and pxiblic policy," guest 
co-editors Daryl Chubin and Sheila Jasanoff . Shelf. 

The first article, by Clark and Ma j one, is a theoretical 
investigation of "The critical appraisal of scientific 
inquiries with policy implications," but hardly discusses 
peer review. Jasanoff "s piece is the most appropriate to 
the vol\me*s topic. It studies peer review of research in a 
regulatory context. There are shorter papers by Porter and 
Rossini; Gillespie, Chubin, and Kurzon; Hamad (see above in 
this section); Kalberer, Roy, Ritvo, and Stossel. 



VI. Quality and Quantity in History of Science and Philosophy 



Baldwin, James Mark^ ed.. Dictionary of Philosophy and 
Psychology , Vol. 2 (N.Y.: Macmillan, 1902), article on "Quality," 
pp. 406-409. File "Quality in Philosophy." 

Bealer, George, Quality and Concept (Oxford: Clarendon Press of 
Oxford University Press, 1982). Wid. L.C. B945 .B3743 Q34 1982. 

Bealer develops a theory of "properties, relations, and 
propositions" which attempts to "unify a great many topics 
central to logic, metaphysics, psychology, and theory of 
language. " 

Bohme, Gemot, Altemativen der Wissenschaft (Frankfurt am Main: 
S>ahr]camp, 1980) . Shelf. 

See especially Chap. Ill, "Antike Alternativen, " which 
discusses Plato and Aristotle. 

Boring, Edwin G., "The beginning and growth of measurement in 
psychology^" in Wool f, ed. (Sec. VI). 
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Brunschvicg, Ueon, l^s Etaoe s de la Phllosophle Matheroatjcrue 
(Paris: Librairies F^lix Alcan, 1912), especially chap. 4, "Le 
mathfenatisme des platoniciens" , pp. 43-69. Chapter 4 in file 
"Quality in Philosophy." Book in Widener Math 66.6. 

This chapter is cited by Koyre in Metapt>Ysj.cs ftpd 
Measurement (Sac. VI) in support of his claim that Plato 
represented two traditions of mathematics: one proto- 
scientific, the other quasi-mystical. 

Brunschvicg, Jjeon, 7^ Rola du PvthaaortsTOe dans 1 'Evolution des 
id^es (Paris: Hermann & Cie., 1937). Widener Phil 1182.10. 

Burtt, Edwin Arthur, The MetaTah vsical Foundations of Modem 
Science (Garden City, N.J.: Doubleday Anchor Books, 1954). Shelf. 

A treatise on quantity, quality, and mathematics in modern 
science. 

Clarkf Joseph T., "The philosophy of science and the history of 
science," in M. Clagett, ed., critical Problems in the History of 
science (Madison, Wise: University of Wisconsin Press, 1959): 
103-140. File "Quality in Philosophy." 

starts from the thesis that ;;> ';' history of science can be 
usefully studied using tlj^ -..f.ytic stmcture of science, as 
opposed to "cumulative" hi.^ic- ical methods per, se. Among 
the topics Clark investigates is Oresme's role in the 
mathematization of physics. 

Crombie, A.C., "Quantification in medieval physics," in Woolf, 
ed. (Sec. VI) . 

Deutsch, Karl W. , et al., "Conditions favoring 

the social sciences," Science . 221 (Feb. 5, 1971): 450-459. File 
"Quality in Philosophy." 

A detailed investigation into the "environmental group 
conditions for creative success in the social sciences," 
with 62 examples drawn from the history of various social 
science fields. 

Edge. David, "Quantitative measures of communication in science: 
A critical ieview," w^ Bf.orv of Science . 12 (1979): 102-134. File 
"Quality in Philosophy." 

A detailed study of those quantitative methods which might 
bo useful in historical studies of scientific comunication. 
Among the quantitative measures discussed are citation, 
co-citation, "trusted assessorship" (personal ^ ^ 

acknowledgements in papers) , co-authorship, teacher-student 
relationships, and "other public traces of informal 
communication." Edge is, on the who!©, critical of 
quantitative methods in the history of science, but does 
conclude that they may be useful as "secondary validation" 
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for qualitative research. 

Fox, Karl A. , Social indicators and Social Theory; Elements of an 
Operational System (H*,3f. : John Wiley & Sons, 1974): Chaps. 1 and 
2. File "Quality in Philosophy*" Book in Gutman Education 
Library H61 .F62. 

Reviews the history (particularly the intellectual Siistory) 
of the "social indicators moyement." Excellent social 
theoretical background. 

Ganz, Carole and Sneed, Joseph D., "Setting priorities: 
Intellectual structure of a discipline," in Carole Cmz, ed., The 
Rise and Fall of Priority Fields , U.S. papers presented as a 
National Science Foundation/European Science Research Councils 
Symposium, Paris, France, Sept. 22-24, 1985. Shelf. 

Sneed is a philosopher of science at the Colorado School of 
Mines who is a principal originator of the "structuralist" 
view of scientific activity, in which the "most important 
products [of that activity] are the mathematical -conceptual 
structures that are used to make. .. [empirical] 
statements. « .A single scientific community is identified by 
its commitment to use a specific conceptual structure to 
deal with a specific range of phenomena" (p. 5). Ganz is a 
philosopher of science working in the Division of 
International Programs at the NSF. The authors develop a 
specific form of content analysis, b^:::.ed on Sneed' s and 
Kuhn*s work, compare the results of the application of this 
technique with cluster maps in co-citation analysis, and ask 
If this approach can "aid in early recognition of scientific 
specialties leading to technologies with social and economic 
value" and "provide some insight into what guides choice of 
research problems and how this choice might be influenced." 
See other, non-annotated, entries under Sneed ii;: this 
section. 

Gerard, R.W. , "Quantification in biology," in Woolf, ed. (Sec. 
VI). 

Guerlac, Henry, "Quantification in chemistry," in Woolf, ed. 
(Sec. VI) . 

Hahn, Roger, A Bibliography of Quantitative Studies on Science 
and its History (Berkeley, Calif, r. University of California at 
Berkeley, Berkeley Papers in History of Science, No. 3, 1980). 
Shelf. 

Holton, Gerald, "Do scientists nec^d a philosophy?" Times fof 
London) Literary Supplement (November 2, 1984): 1231. File 
"Holton Papers." 

Suggests that the *^£malysi& of current dimensions of 
scientific innovation" should begin to draw more upon ethics 
than it has in the past, given the fundamental moral 
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problems with which aclence is Involved in the contemporary 
world. 

Kevles, Daniel J., "The health of science," Harpers (Aug. 1979): 
26-31. File "Quality in Philosophy." 

Argues that th& period 1945-1970 in American science was 
anomalous, that American science is healthy, current 
declining budgets and the warnings of eminent scientists 
notwithstanding, and that continued emphasis on supporting 
the best students and researchers will keep it strong. See 
also Nye, and Paul (Sec. VI). 

Koyre, Alexandre, Metaphysics and Measurement (Cambridge, Mass.: 
Harvard University Press, 1968). Shelf. 

A central work in the field, and relevant to any study of 
the philosophical foundations of, science indicators. See 
especially Chaps. 1 and 2: "Galileo and the scientific 
revolution of the seventeenth century" (pp. 1-15) and 
"Galileo and Plato" (pp. 16-43). 

Kuhn, Thomas S., "The function of measurement in modem physical 
science," in Wool f, ed. (Sec. VI). 

Kyburg, Henry E. , Jr., Philosophy of Science: A Formal Approach 
(N.Y.: Macmillan, 1968), Chap. 3, "Quantities," pp. 55-86. 
Shelf. 

Although a "formal approach," Kyburg recognizes a continuum 
of quantitization of measurement among the different 
sciences. "Sciences that employ ... (fully quantitative 
judgements) are no doubt more powerful and more useful (they 
provide quantitative predictions) than sciences that do not; 
but we should not regard them as different in basic 
characteristics" (p. 85) . 

Lasswell, Harold D. , "The qualitative and quantitative in 
political and legal analysis," in Lerner, ed. (Sec. VI). 

Lazarsfeld, Paul F. , "Notes on the history of quantification in 
sociology — trends, sources, problems," in Wool f, ed. (Sec. VI). 

Leontief, Wassily, "The problem of quality and quantity in 
economics," in Lemer, ed. (Sec. VI). 

Lerner, Daniel, "Introduction, on quantity and quality," in 
Lerner, ed. (Sec. VI). 

A historical account of the topic and an introduction to the 
papers in the volume. 

Lerner, Daniel, ed. , Quantity and Oualitv: The Havden Colloquiuin 
on Scientific Method and Concept (N.Y.: Free Press of Glencoe, 
1961). Widener S 308.36. 
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See Lemerr Welsskopf, Stevens, Rosenblith, Lasswell, 
Leontief , and Spengler in Sec. VI, and Keaeny in Sec. VIII. 

Morman, Edward T« ^ •^Citation analyeis etnd the current debate over 
quantitative y^^" - in the social studies of science," 45 
fSocietv for . : .1 , Studies of Science^ Newsletter , 5 (Summer 
1980): 7-13. Fil^ "Citations. 

Intellectual history of the debate over the appropriateness 
of citation and co-citation analysis in the history of 
science. Focusses on Price, Garfield, and the Coles on the 
one hemd, zaid Edge on the other (though other authors are 
discussed). Useful set of references. 

jHulkay, Michael, "Applied philosophy and philosophers* practice," 
science^ Technology. & Human values . ^ (Winter 1981) : 7-15. 

Critiques papers in the previous issue of STHV ^ which was 
partly on applied philosophy of science. (See paper by 
Shrader-Frechette in Sec. VI.) The author is a sociologist 
and his theme, put simplistically, is that both scientists 
and philosophers practice within social groups, that their 
practice differs, and that these differing practices affect 
epistemological or analytical approaches. In order to 
"influence others* actions," applied philosophers "might 
have to engage actively in the social life of their subjects 
[scientists] .. .instead of relying almost exclusively on 
analysis of the general structure of scientsts* intellectual 
accomplishments . " 

Myerson, Emile, Identity and Reality , trans. Kate Loewenberg 
(N,Y.: Dover :5ublicat ions, 1962). Shelf. 

Of most immediate interest is Chap. 10, "Non-mechctnical 
theories pp. 323-353, an account of the evolution of 
"Aristotelian science." 

Naydler, Jeremy, "The poverty of Popperism," The Thomist , 46 
(Jan. 1982) : 92-107. 

A critique of one variant of logical positivism from the 
point of view of a neo- (medieval) realist. 

Naydler, Jeremy, "The regeneration of realism and the recovery of 
a science of qualities," International Philosophical Quarterly , 
21 (June 1983): 155-172. File "Quality in Philosophy." 

An account of Medieval realism, its Aristotelian and 
Platonic roots, and its supposed degeneration into 
"nominalist modern science • 

Nye, Mary Jo, "Scientific decline: quantitative evaluation 
enough?" Isis, IS (1984): 697^'7QB. rile "Qviality in Philosophy." 
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A study of .nineteenth centuiry attempts to quantify the 
supposed decline of French science. Draws parallels with 
the current situation in the US. Considerable relevance to 
the modern science indicators movement. See also Kevles, 
and Paul (Sec. VI) . 

Paul, Henry W., The Sorcerer's Appr entice; The French Scientist's 
Tmaae of German Science; 1840-1919 (Gainesville, Fla.: University 
of Florida Social Sciences Monograph No. 44, 1972). Shelf. 

A case study illustrating the semi-subjective nature of the 
evaluation of the quality of scientific research. See also 
Nye, and Kevles (this Sec.) . 

Ravetz, Jerome R. , Scientific Knowled ge and Its Social Problems 
(Oxford: The Clarendon Press of Oxford University Press, 1971), 
Chap. 10, "Quality control in science." File "Measuring Output." 
Book at Cabot Q175 .R3. 

Argues strongly for qualitative approach to evaluating 
scientific research, though does not explicitly employ the 
terms "qualitative" and "quantitative." The author observes 
that "It is impossible to dssign a simple set of routine 
tests, by which one could assign some numerical marks to a 
scientific problem, and then grade it on a linear scale. Nor 
indeed would it be feasible to erect a formal system of 
categories of quality, and train up a corps of expert 
assessors. . .to operate in their framework. For the 
techniques are so subtle, the appropriate criteria of 
adequacy and value so specialized, and the materials so 
rapidly changing, that any fixed and formalized categories 
would be a blunt and obsolete instrument as soon as it were 
brought into use. The result of this special characteristic 
of science is that if there are to be truly expert 
assessments of quality of work, they must be made by a 
section of those who are actually engaged upon that work." 

Rescher, Nicholas, Scientific Progress ; A Philosophical Essay on 
the Economies of Research in Natural Science (Pittsburgh, Penn. : 
University of Pittsburgh Press, 1979). Cabot Q 180.55 .E25 R47 
1978b. 

See also a largely positive review by Philip Morrison in 
Rnipnf^fin American. 240 (May 1979): 37-40. (File "Quality 
in Philosophy" under Bescher. ) Morrison describes the book 
as follows: "The philosopher-analyst author has sought to 
quantify the almost indefinable and to project its future 
from a few empirical results and a boxful of argument, based 
on a sweep of thought from. . .Leibniz, .. .Peirce, .. .and 
Reymond to ...Feynman and. ..[D. de S.] Price." 

Roger, Jacques, "Science, nature, quality," pjogenes. No. 88 
(Wint4r 1974): 69-76. File "Quality in Philosophy." 

A review of thfe philosophic bases of Aristotelian and modern 
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physics / with particular reference to quality and quantity. 

Rosenblith^ Walter A. ^ ••The quantification of the electrical 
activity of the nervous system^ •• In Lerner^ ed. (Sec. VI). 

Schware, Robert, Quantification in the Histo ry of Political 
Thought! Toward a Qualitative Approach (Westport, Conn: Greenwood 
press, 1981). Widcner L.C. JA.81 S347. 

An examination of quality and quantity in the political 
thought of philosophers from Plato to Weber. Separate 
chapters on Bacon, Hobbes, and Mill. 

Shrader-Frechette, Kristin S., "Technology assessment as applied 
philosophy of science," .S cience. Technolo gy > & Human Values, 6 
(Fall 1980) : 33-50. 

The author elaborates upon the following statement: "A 
philosopher of science can contribute to technology 
assessment and hence ultimately to the guidance of 
technology-related public policy on at least two levels: 
that of conceptual and methodological analysis ^ and that of 
concrete applications^ where the concepts and methods are 
employed in specific contexts." 

Shryock, Richard H. , "The history of quantification in medical 
science," in Wool f, ed. (Sec. VI). 

Sneed, Joseph D. , "Philosophical problems in the empirical 
science of science: A formal approach," Erkenntnis . 10 (1976): 
115-146. File "Quality in Philosophy." 

See Ganz and Sneed in this section. 

Sneed^ Joseph D. , The Logical Structure of Mathematical Physics . 
2nd ed. (Dordrecht^ The Netherlands and Boston: Reidel^ 1979). 
MIT Science Library QC20 .S671 1979. 

See Ganz and Sneed in this section. 

Spengler^ Joseph J.^ "On the progress of quantification in 
economics^" in Woolf^ ed. (Sec. VI). 

Spengler^ Joseph J. ^ "Quantification in economics: Its history/' 
in Lerner, ed. (Sec. VI). 

Stam^ David H. ^ "Concluding unscientific postscript: Reflections 
on selectivity from a nontechnical perspective/' in K.S. Warren^ 
ed.^ Select ivitv in Information Systems : 166-170. (See Sec. 
II.) 

A short think-piece on selecting high-quality literature. 
The author is director of the New York Pxablic Library's 
research division. 
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Stevens, S.S., "The quantification of sensation," in Lerner, ed. 
(Sec. VI) . 

Weisskopf, Victor F. , "Quality and quantity in quantum physics," 
In Lerner, ed. (Sec. VI) . 

V7ilks, S.S., "Some aspects of quantification in science," in 
Woolf , ed. (Sec VI) . 

Review of quantification and measurement in the history and 
philosophy of science. 

Woolf- Harry, ed. , Quantification! A History of the Meaning of 
Measureiti^.nt in the Natural and aocial Sciences (Indianapolis, 
Ind.: Bobbs-Merrill , 1961. Widener S 840.95. 

Papers read at a conference held at the offices of the 
Social Science Research Council Nov. 20-21, 1959. The 
conference was affiliated with the Joint SSRC-NRC Committee 
on the History of Science. See Wilks, Crombie, Gerard, 
Boring, Lazardcfeld Kuhn, Guerlac, Shryock, and Spengler, ii 
Sec. VI. for titles of papers. 

Zandvoort, Henk, "An extension of Sneed's reconstruction of 
classical particle mechanics to complex applications, and an 
alternative approach to special force laws," Erkenntnls . 18. 
(1982): 39-63. File "Quality in Philosophy. " ■ 

See Ganz and Sneed above, in this section. 

Zandvoort, Henk, "Comments on the notion 'Empirical claim of a 
specialization theory net' within the structuralist conception o 
theories," Erkenntnis . 18 (1982): 25-38. File "Quality in 
Philosophy." 

See Ganz and Sneed above, this section. 

Zuckerman, Harriet, "Theory choice and problem choice in 
science," Sociological Induirv . 48 (1978): 65-95. File 
"Sociology of Science." 

Reviews "current sociological and philosophical work bear in 
on cognitive change [in science]." Finds that scientists' 
behavior corresponds but little to "epistemological 
prescriptions of how they should behave," but "corresponds 
more than one might suppose to often stated methodological 
precepts which give primacy to assessed significance and 
feasibility of solution." 



VII. Education 

Bayer, Alan E. and Folger, John, "Some correlates of a citation 
measure of productivity in science," Sociology of Education. 39 
(1966): 381-390. File "Citations." COZZENS. 
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Authors* abstract: "The Science Citation Index provides an 
easy vay to derive criterion measures of scientific 
acconpllshment. KTeaaures derived from citation counts, the 
principle criterion , have high face validity. These 
criterion measures are found to have a low but positive 
correlation with the quality of scientists' graduate 
education [based on Cartter report} and no relation to his 
measured I.Q. score." 

Beyer, Janice M., and Snipper, Reuben, "Objective versus 
subjective indicators of quality in graduate education," 
Sociolocrv of Education , ^7 (Fall 1974): 541-557. File 
"Education." 

Investigates the difficulty with which the same quantitative 
measures are applied across different scientific 
dlBciplines. Uses "both original data and data from 
published sources" to examine "the relationship of the rated 
quality [in the Cartter report] of university departments to 
other pos;3ible quality indicators in two physical and two 
social sciences." The original data is based on 
questionnaires sent to faculty members, which ask them to 
name scholars whom they vould like to hire for their 
departments. That is, the "other possible quality 
indicators" are "subjective." One important conclusion is 
that "objective measures might substitute for the collection 
of reputational data on quality" only if "a large enough 
range of data is collected to cover all of the measures that 
might be important in discriminating within different fields 
and disciplines. Since different objective indicators were 
the most important discriminants or correlates of quality in 
different fields, there is no assurance that any particular 
list of variables would be adequate to predict quality 
across all fields and disciplines." See also papers by 
Knudsen and Vaughan and by Hoore (Sec. VII.). 

Cartter, Allan M. , An Assessment of Quality in Graduate Education 
(Washington, D.C.: American Council on Education, 1966). Kennedy 
School of Government Library LB2371 .C35. 

Cartter uses expert opinion to rank graduate programs in 
various disciplines. The report is based on the assumption 
that "the higher the degree of agreement among expert 
witnesses, the more likely It Is that the opinion represents 
a fact.^ See also critiques by Knudsen and Vaughan and by 
Bayer and Snipper (both in this section) . 

Dawes, Hobyn M. , "A case study of graduate admissions: 
Application of three principles of human decision mzdcing," 
American Psychologist , 26 (April 1971): 180-188* File 
"Education." 



Investigates the possibility of screening out clearly 
unacceptable applicants to graduate departments using 
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mathematical models of the observed decision-making 
processes of admission committees. 

Knudsen, Dean D. and Vaughan, Ted R. , "Quality in graduate 
education: A re-evaluation of the rankings of sociology 
departments in the Cartter report," The Ameri can Sociologist 
(Feb. 1969): 12-19. File "Education." 

Criticizes the Cartter report for its "subjective 
evaluations" and attempts to disceini the correlation 
between these and more "objective indicators." The 
objective indicator used is publications and the authors 
find that the ranking of the best departments so indicated 
corresponds closely with that of the Cartter report, while 
the ranking of departments of lesser quality does not 
correlate well. See also Moore and Beyer and Snipper (Sec. 
VII . ) . 

Moore, William J., "The relative quality of graduate programs in 
economics, 1958-1972: Who published and who perished," Western 
Economic Journal (1973): 1-23. File "Education." 

Uses rather elaborate mathematical models to find that 
publication rates are highly correlated with the Cartter 
report's rankings. See also Knudsen and Vaughan, and Beyer 
and Snipper (Sec VII.). 

National CCr/^ v' for Education Statistics, Three National 
AssessmentL- Science; Changes in Achievement. 1969-7 7 (Selected 
Results froi?> ^np Third National Assessment of Science) . (Denver, 
Col.: Education Commission of the States, June 1978, Science 
Report 08-S-OO) . File "Education." 

Siimmary of major findings: 1/ Overall achievement in science 
Is not declining nearly as quickly as it previous had been; 
2/ The achievement of seventeen year-olds, however, 
continues to decline; 3/ The achievement of students in 
"extreme-rural communities" is improving; 4/ "A gap 
continues to exist in the achievement levels of whites and 
blacks" and "The achievement level of males at each age was 
higher than that of females." 

Schatz, Gerald S., "Science for non-science majors: Problems of 
quality and quantity," News Report (of the National Academy of 
Sciences), (March 1982): 3-S. File "Education." 

Reviews a report by the National Research Council entitled 
Science for Non-Specialists: The College Years , published in 
1982. Advocates improvement in non-specialist science 
programs . 

Smith, Richard and Fiedler, Fred E., "The measurement of 
scholarly work: A critical review of the literature," Educational 
Record . 52 (Summer 1971). File "Measuring Output." 
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Focuses on the strengths and weaknesses of a survey of 
faculty conducted by the American Council of Education in 
1964 to determine the quality oi' academic departments. 
States that "Most ratings of individual faculty members are 
based on publications," and then goes on to elaborate. 

Solso, Robert L. , "Recommended readings in psychology during the 
past 17 years," American Psvchologlst , 26 (Dec. 1971): 1083-1084. 
File "Education." 

Based on analysis of reading lists. 



VIII. Issues involving quantity and crualltv in particular 

disciplines, including papers on social indicators. (See 
also entries In Sec. VI, especially volumes edited bv Woolf 
and bv Leiner ^ 



A. Social Indicators 

Abom, Murray, "The short and happy life of social indicators at 
the National Science Foundation," Items (Newsletter of the Social 
Science Research Council), 23, No. 2/3 (Sept. 1984). File 
"Disciplinary." 

Historical account of the program on social indicators, 
which ran from 1971 through 1982, review of perspectives on 
causes for its demise, and list of major projects funded. 

rohmstedt, George W. and Borgatta, Edgar F., eds.. Social 

o asurement: Current Issues (Beverly Hills, Cal.: Sage, 1981). 
Widener L.C. HA29 .S638. 

The papers in this volume focus on tecl^nical, mathematical 
issues. 

Car ley, Hichael, Social Measurement and Social Indicators: Issues 
of Policy and Theory (London: George Allen & Unwin, 1981). 
Widener L.C. HN25 .C36. 

Ofti^n takes a historical approach, frequently discussing the 
literature on social indicators. Chapter titles: 
Introduction to the scope of social indicators; Definitions 
and dimensions of social indicators; Social indicator 
systems; Social theory and models; Social indicators and the 
policy-making process; National social reporting (focusing 
on the U.K. and U.S.); Social indicators at the urban level. 

Duncan, Otis Dudley, Notes on Social Keasvf.rement: Historical and 
Critical (N.Y.: Russell Sage Foundation, 1984). Widener L.C. 
HM253 .D86 1984. 

Emphasis on historical evolution of social measurement 
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(beginning with the Greeks) . Includes analysis of the 
development of quantitative methods in the social sciences. 
Table of contents includes abstracts of chapters, whose 
titles are: Inventing social measurement; Historical 
metrology; More inventions (e.g. modem ideas about chance) ; 
On scales of measurement; Measurement: the real thing; 
Psychophysics; Psychometrics ; Social measurement: 
predicaments and practices. 

Fox, Karl A. , Social Indicators a nd Social Theory; Elements of an 
Qoerational System (N.Y. : John Wiley & Sons, 1974): Chaps. 1 and 
2. File "Quality in Philosophy." 

Reviews the history (particularly the .'.ntellectual history) 
of the "social indicators movement." 

Francis, Walton, "What social indicators don't indicate," 
Evaluation . 1 (1973): 79-83. File "Disciplinary." 

Analysis of several basic difficulties with developing and 
interpreting quantitative indicators of quality of life. 

Lee, Trevor and Marans, Robert W. , "Objective and subjective 
indicators: Effects of scale discordance on interrelationships," 
Social Indicators Research . 8 (1980): 47-64. File 
"Disciplinary. " 

Investigates and attempts to offer explanations for weak 
relationships between objective and subjective measures of 
quality of life. "The hypothesis that the relationship 
between the objective and subjective measures is stronger 
among individuals whose view of neighborhood size is in line 
with the relatively large territorial base for objective 
crime statistics is tested and found to be correct." 

Seidman, David, book review of Ben-Chieh Liu, Quality of Life 
Indicators in U.S. Metropolitan Are as: K Statistical Analysis 
Tn^.Y.: Praeger, 1976), in Social Indicators Research. 4 (1977): 
97-106. File "Disciplinary." 

This is an important piece, together with the reply by Liu 
and the rejoinder by Seidman, included with the review at 
the same location in the file. Seidman 's "principal 
contention here is that the production of composite indices 
of quality of life rests upon, at best, v&ry shaky 
foundations." He goes on to make specific technical 
criticisms of the composite indicators that Liu uses, and 
concludes that "In view of the gross shortcomings of Quality 
of Life. . . . the attention it has received can only be 
considered unfortunate. An atheoretical , ad hoc effort, it 
is presented in such a way as to lead the unwary to think 
that some scientific basis underlies the rankings of the 
[Standard Metropolitan Statistical Areas]." 

Shaw, Anthony, "Defining the quality of life," Hastings Center 
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Report . 7 (Oct. 1977): 11. Pile "Disciplinary." 

Research note on symbolic representation of quality of life 
using a "formula without numbers." 

O'Neill, "S.>ae relevant policy uses of the national longitudinal 
surveys," Social Indicators Newsletter ^ No. 18 (Sept. 1983). 
File "Disciplinary." 

Examines the actual and potential applications of the 
national Longitudinal Survey of Labor Market Experience. 
The major categories of use which O'Neill discusses are 
unemployment issues, women's labor force, and aging and 
retirement. (O'Neill references a study of her own prepared 
for the Department of Labor, "Women and the labor market: A 
survey of issues and policies in the United States.") 
Potential uses mentioned include health, education, and 
military manpower. 

"Poverty is more than being flat broke," interview with Peter 
Townsend, Professor of Social Policy at the University of 
Bristol, New York Times , April 11, 1982. File "Disciplinary." 

Townsend discusses his "deprivation index," which incluCoss a 
variety of indicators that cannot be measured in monetary 
terms. 

TAiite, Howard D. , "A cocitation map of the social indicators 
movement," Journal of the American Society for Information 
Science, M (1983): 307-312. File "Disciplinary." 

Review of the social indicators literature using cocitation 
techniques to organize this literature. "The [cocitation] 
map conforms well with independent reviews of the SI 
literature, actually illustrating some of the judgements 
they contain." Includes an excellent, and effectively 
annotated, bibliography. 



B. H istory 

Aydelotte, Quantification In History (Reading, Mass. ; 
Addison-'Wesley, 1971). Widener H 4539.71. 

Argues for a quantitative appraoch to the study of history. 
Case study of modern British history. 

Servos, John W. , "Trends of chemistry," review of Arnold 
Thackrayf et al.. Chemistry in America, 1876--1976> Historical 
Indicators (Boston: Reidel, 1985), Science , 230 (November 15, 
1985): 800. File "Disciplinary." 

The book reviewed amasses a great deal of quantitative data 
which apparently indicates that chemistry has been in 
decline for some time, relative to other disciplines. The 



64 



authors of the book are dedicated to quantitative history 
and do it well, but Servos wishes "they had been less 
ruthless in suppressing their interpretative instincts and 
more forthcoming in discussing those issues which... give 
those data meaning." 

Thackray, Arnold, "Measurement in the historiography of science," 
in Elkana, et al., eds.. Toward a Metric of Science (N.Y.5 Wiley, 
1973): 11-3 0. 

Investigates the value of quantitative methods for the 
history of science and for contemporary attempts to measure 
science. 



C. Health 

Baumahn, Barbara, "Diversities in conceptions ^--.'.alth and 
physical fitness," Journal of Health an d Human y 4iavior , 2. 
(Spring 1961): 39-46. File "Disciplinary." 

Discusses various conceptions of physical health. 

Comroe, Julius H., Jr. and Dripps, Robert D. , "Scientific basis 
for the support of biomedical science," Science . 3-92 (April 9, 
1976): 105-111, File "Disciplinary." 

See Garfield entry, "How can we prove the value of basic 
research?" (Sec. II.) for discussion. 

Culyer, A.J., Measuring Health: lessons for Ontario (Toronto: 
University of Toronto Press, 1978): Chap. 2, "Indicators and 
indexes: An overview." File "Disciplinary." 

A theoretical approach, focusing od events in Canada. 

Grogono, A.W. and Woodgate, D.J., "Indejj for measuring health," 
The Lancet (Nov. 6, 1971): 1024-1026. File "Disciplinary. 

"'A health index is described which expresses health 
numerically between 0 and 1. Each patient scores 0, 1/2, or 
1 for each, of ten 'questions' relating to well-being." 

Leeper, Peppe?:, "Which disease? What vaccines? How to set 
priorities," '^ex^; '? Report (of the National Academy of Sciences), 
35" (Dec. 1984-J^xi. 1985): 4-11. File "Disciplinary." 

Includes section on "n on- quantitative fvictors." 

Marshall, Eliot, "Psychotherapy faces test of worth," Science, 
207 (Jan. 4, 1980). File "Disciplinary." 

Discusses the politics of the psychotherapy community 
attempting to measure the benefits of its treatment in order 
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to qualify for Medicare reimbursement. 

Hiller, James E., ^An indicator to aid management in assigning 
program priorities^" ^blic health Reports , 85 (August 1970): 
725-731. File ''Disciplinary." 

Develops a quantitative indicator of the effect of public 
health progr2uns cn the health of & population. 

United States Congress, Office ox Technology Assessment ^ 
Medicare's Prospective Payment System; Strategies for Evaluating 
Cost, Oualitv. and Medical Technologrv (Washington, D.C.: USGPO, 
October 1985). Report Brief in File "Discip] '--^ary. " 

The report includes both substantive and methodological 
dimensions. 



D* Policy Evaluation 

Glaser, Edward H. and Blacker, Thomas E., "outline of questions 
for program evaluators utilizing the clinical approach," 
Evaluation . 1 (Fall 1972): 56-60. File "Dijciplinary . " 

Basic summary of policy evaluation. One short section on 
"Methods of Measurement," discussing appropriateness of 
"objective" or "sxibjective" techniques. 

House, Peter W. and Shull, Roger D. , The Rush to Policy; Using 
Analytic Techniques in Public Sector Decisionnaking (privately 
published, circa 1982). R. Stowe*s library. 

The authors, who at the time of publication were in the 
Division of Policy Research and Analysis at the National 
Science Foundation, describe various analytical techniques 
and specify their strengths and weaknesses. Included are 
discussions of the the interaction between political and 
analytical components of policy-making. Chapter titles: 
Quantitative techniques; Benef it*-cost analysis; Risk 
analysis; Decision analysis; Practicability, relevance, and 
compatibility c£ quantitative decision models; 
Constitutional and bureaucratic framework; Theory and 
process of decisions; Policy in practice; Making room for 
analysis in the political process • 

Housc^^a^ Tom R. , Jr., "The behavioral sciences impact- 
effectiveness model," in P.H. Rossi and W. Williams, eds.. 
Evaluating Social Programs: Theory, Practice, g ivl P olitics (N. Y . : 
Seminar Press, 1972): 51-65. Paper in file "DitiiuXplinarj?. " Book 
at Widener Soc 562.716.10, 

Summary of techniques for and approaches to policy 
evaluation. Discusses experimentation and "preexperimental 
designs" for evaluating policy in some detail. Although 
problems with experimental evaluation of policy are 
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discussed, Houston concludes that it is "a powerful model 
for evalua'cion. " 

Patton, Michael Quinn, Qualitative T^ayiatlop Methods (Beverly 
Hills, Cal.: Sage, 1980). H62 .P3218. 

Includes discussion of the reletiva merits of qurintitative 
and qualitative approaches. 

Yin, Robert K. , et al., "The difference that criiality ma)ces: The 
case of literature reviews," Sociolog ical Methods & Research , 1 
(Nov. 1976) I 139-156. File "Measuring Output." 

Investigates and attempts to operational ize two intrinsic 
measures of quality in research on innovation in public 
policy: "tlie degree to which both the intervention process 
and its outcomes were measured with reliable and valid 
instr rnents," and "the degree to which the evaluation ^^iidy 
followed the paradigm for experimental research..." This is 
not a study of literature reviewing per se (cf. Light and 
Pillemer, Cook and Leviton [Sec. II.]). It is rather a 
study of how to measure the quality of individual research 
projects which might be included in reviews. 



miscellaneous 

Brooks, Harvey, "The military innovation system and the 
qualitative arms race," Daedalus . 104/ No. 3 (Summer 1975): 
75-97. Shelf. 

Discusses the impact of technological innovation on the 
"world military balance." concludes in part that "the most 
promising lines of action for controlling the qualitative 
arms race probably lie in mutually agreed limitations on 
testing, including limits on the numbe- of permissable 
launchers, and on a comprehensive nuclear-test ban." 

De Millo, Richard A., et al., "Social processes and proofs of 
theorems and programs," communicat ions of the ACM. Zl (May 19/9): 
271-280. File "Disciplinary." 

A philosopii^ ..ally reflective discussion of the dif fi'^ulties 
involved with mathematically proving the correctness of 
computer programs. The authors state that "...in the end, 
it is a social process that determines whether 
mathematicians feel confident about a theorem— and we 
believe that, because no comparable social processes can 
take place among program verifiers, [a rigorous approach to] 
program verification is bound to fail. 

Glaser, Barney G. and Strauss, Anselm L., The Discovgry pf 
Grounded Theory: str? <-^q^ fr^r- Qualitative Research (Hawthorne, 
N Y ? Aldine, 1967) . Gutman Education Library HM48 .G43. 
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Guide for researchers in "the social sciences, with some 
reflection on the relationships between and complementarity 
of quantitative and qualitative data. 

Holmfield, John D. , "Science indicators and other indicators: 
Some user observations," as (SQc^^^y fn^c Social Studies of 
Science) Newsletter . 2 (Fall 1978): 36-43. File "Science 
Indicators . " 

The author is on the staff of the House Committee on Science 
and Technology, "in this paper the focus will be on the 
lessons and insights that have been learned from the 
development and use of indicators in a number of other 
fields and their applicability to SI." Discusses various 
sorts of indicators (e.g. of water quality, housing 
quality) . 

Joneis, Lyle V., "The assessment ©f scholarship," in E.H. 
Loveland, ed. , Measurin g, the Ha irH-to-Measure . No. 6 in the series 
New Directions in Proarai" Evaluat-Mop (San Francisco: Jossey-Bass, 
1980). File "Measuring output." 

Reviews several criteria of the quality of an academic 
researcher's work: number of publications, citation 
indexing, peer judgements, and awards and honors. Includes 
a brief section entitled "Evaluating scholarship in the 
performing arts, useful sxunmaries of the literature on each 
of these topics. 

Juster, F. Thomas, "Alternatives to GNP as a measure of economic 
progress," in U.S. Congress, Joint Economic Committee, U.S. 
Economic Growth f r-nm ; io76 to 1986; Prospects. Problems, and 
Patterns . Vol. 10, 7!^^ ntialitv of EcQnomic Growth (Washington, 
b.C: USGPO, May 20, 1977): 12-24. File "Disciplinary." 

Suggests improvements in the measures derived from the 
National Income and Product accounts, but also concludes 
that "significant dimensions of both economic growth and 
economic welfare are. basically perceptual rather than 
objective, and no transformation or modification of the 
present structure of Accounts is able to incorporate these 
types of data." This volxaue also includes a paper by N. E. 
Terleckyj on "Economic growth and the quality of life." 

Kesneny, John G» , "Mathematics without numbers," in Lerner, ed. 
(Sec. VI) . 

Discusses semi-quantitative mathematical approaches to ti.'z 
social sciences. 

Hogerson, William P., "Reputation and product quality," 
California Institue ot Technology, Division of the Humanities and 
Social Sciences, Social Science working Paper No. 330, July 1980. 
File "Disciplinary." 
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Economic analysis of a market characterized by "quality 
uncertainty" with respect to products. 



IX. SQcioloav of science f including sc ientific comnunlcation) 

Ben-David, Joseph, "The profession of science and its powers," 
Minerva , 10 (July 1972): 362-383. File "Sociology of Science." 

Concludes that "scientists and the scientific community can 
only participate in the still inadequately understood 
processes of using scientific research for the solution of 
practical problems. Claims for exclusiveness of their 
expertise, for an exclusively professional control over the 
allocation of funds and the execution of projects with such 
ends in view might bring short-run benefits for science. 
They are unlikely to serve the long-range objective of 
making research an increasingly more useful tool for man." 

Chubin, Daryl E. , "State of the field: The conceptualization of 
scientific specialties," The socio] nair.;^! Quarterly. 12 (Autumn 
1976): 448-476. File "Sociology of Science." 

The paper "analyzes the phenomenon of research specializtion 
in science. The format consists of two ' sections . The first 
features a state-of-the-art review of evidence from 
so-called specialty case studies on definitions, measurement 
strategies, and representations of the relations in which 
small groups of researchers cohere. In the second section, 
a theoretical perspective on the development of specialties 
is formulated. This perspective ... suggests ... that core 
researchers derive innovations from the margins of their 
■itiecialty. It is further hypothesized that both maintenance 
and realignment of social structures, i.e. communication and 
status configurations, depend on intellectual events that 
occur in the course of normal scientific progress." 

Cole, Stephen, "Age and scientific performance," American Journal 
nf SoclolScTV . 84 (Jan. 1979): 958-977. File "Sociology of 
Science." (See also article in this section by Gillmor.) 

The authors conclude that "The long-standing belief that age 
is negatively associated with scientific productivity and 
creativity is shown to be based upon incorrect analysis of 
data." 

cole, Stephen, "The growth of scientific knowledge: Theories of 
deviance as a case study," in L.A. Coser, ed. , The ydea of Socia l 
R i:Tneture; Papers in P»r.nr of Robert K. Merton (N.Y.: Harcourt 
Brace Jovanovich, 1975) . Paper in File "Sociology of Science"; 
book in Social Relations Library HM24 .134. 

Uses Merton's theory of deviance (social structure and 
anomie) and subsequent literature on deviance "as a research 
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sito to explore the social processes through which knowledge 
grows and ideas change." Cole Ibuilds on the work of Thomas 
Kuhn, who "has concluded that, ultiinataly, the process 
through which ideas change and develop can be understood 
only through sociological analysis." The author outlines 
"some of the problems raised by 'i^uhn and then, using the 
field of deviance research as an example," illustrates "how 
we might go about empirically investigating a few of these 
problems." This investigation relies on citation analysis. 

Cole, Stephen, "Professional standing and the reception of 
scientific discoveries," American Journal of Sociolo<JYr 21 (Sept. 
1970). File "Sociology of Science." 

Tests the hypothesis that "if the Matthew Effect were to 
operate, the reception of papers of equal quality should be 
influenced by the location of their authors in the 
stratification system." Measures quality with citations. 
Cole finds that, for the most part^ the hypothesis is not 
supported by the data. 

Cole, Stephen and Cole, Jonathan R. , "Scientific output and 
recoCTrition: A study in the operation of the reward system of 
science," American Sociological Review , 32 (June 1967): 378-390. 
File "Citations." COZZENS. 

Studies "The relationship between the quantity and quality 
of scientific output of 120 university physicists. Although 
these two variables are highly correlated, some physicists 
produce many papers of little significance and others 
produce a few papers of great significance." Quality is 
measured by citations, using SCI . 

Elzinga, Aant, "The Swedish science discussion 1965-1975," Social 
Indicators Research , 7 (1980): 379-399. File "Sociology of 
Science." 

Author's abstract: "During the 1960 's, in Sweden. ..GNP was 
taken as an indicator of scientific growth. Today the 
science policy discussion is more centered around the 
question of political direction of science in accordance 
with social goals. ..This change of science policy doctrine 
reflects important modifications in the social conditions 
for the production of scientific Jcnowledge." The paper 
discusses these propositions in light of future studies, 
controversy over Bci^noe, and trade union interest in 
issience policy. 

Fisher, Charles S., "The last invariant theorists: A sociological 
study of the collective biographies of mathematical specialists," 
Archives Europennes de Socjolocrie , 8 (1967): 216-244. File 
"Sociology of Science." (See also entry under Chub in in Sec. II. 
on collective biography.) 

The author takes "the perspective of the sociology of 
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knowledge" and tries to "describe the disappearance of a 
mathematical theory. Although accepting that the 
"intellectual content of a theory" is the basis of "one way 
of describing occurences within science," he chooses to view 
science as "as an activity carried on by the men who create 
the scientific ideas." 

Garvey, William D. and Griffith, Belver C. , "Scientific 
communication: Its role in the conduct of research and creation 
of knowledge," /American Psychologist . 26 (April 1971): 349-362. 
File "Sociology of Science." 

Discusses problems with scholarly c >iiimunication in the field 
of psychology and possible solutions to these problems. 

Gilbert, G. Nigel, "Referencing as persuasion," Social S tudies of 
Science . 2, No. 1 (Feb. 1977): 113-122. File "Sociology of 
Science." COZZENS. 

The author considers "scientific papers as 'tools of 
persuasion. ' A scientist who has obtained results which he 
beleives to be true and important has to persuade the 
scientific community .. .to share his opinions of the value of 
his work. For it is only when some degree of consensus has 
been achieved that his research findings will be transformed 
into scientific knowledge." 

Gilbert, G. Nigel, "The transformation of research findings into 
scientific knowledge," Social Studies of Science. 6 (1976): 
281-306. File "Sociology of Science." COZZENS. 

The paper is concerned with "the process whereby a 
scientist's research findings are transformed into 
accredited factual ]cnowl edge. . .with the 'context of 
justification' .. .In contrast to the epistemological concerns 
of the philosophers," the author considers "only the 
procedures actually used by the natural scientists to decide 
on the validity of claims to scientific knowledge." 

Gillmor, C. Stewart, "Aging of geophysicists, " Eos, 65 (May 15, 
1984): 353-354. File "Sociology of Science." (See also article 
in this section by S. Cole.) 

Does not look at productivity, but rather at the causes for 
the aging of the population of geophysicists. 

Gustin, Bernard H. , "Charisma, recognition, and the motivation of 
scientists," American Journal o f Sociolocrv. 78 (March 1973): 
1119-1134. File "Sociology of Science." 

"Desire for recognition is a theoretically and empirically 
inadequate key to the motivation of scientists. A large 
proportion of the scientific community publishes very 
little. The elite, prolific scientists are neither 
dependent on nor rewarded for the publication of their 
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research^ and most work by average scientists is hardly ever 
acknowledged, much less rewarded, with citations and prizes. 

Charisma is proposed as a complemetary basis for an 
explanation of scientific motivation... 

Merton, Robert K. , Foreword to E. Garfield, Citation Indexing: 
Its Theory and Application in Science. Technology, and 
Humanities (John Wiley & Sons, 1979; reprinted in Current 
Contents ) , File "Citations." 

Review of citation analysis from the point of view 
of sociology of science. 

Merton, Robert K. , "The Matthew Effect in Science," Science . 159 
(Jan. 5, 1968): 56-63. File "Sociology of Science." 

Suggests 1:hat those scientists who achieve early success 
tend, for that reason, to enhance their chances of future 
success. That is, rewards in the scientific community are 
allocated preferentially to those who have already gained a 
measure of prominence. 

Mulkay, Michael, "Applied philosophy and philosophers' practice," 
Science. Technolog y, & Hum an Values , (Winter 1981): 7-15. 

Critiques papers in the previous issue of STHV , which was 
partly on applied philosophy of science. (See paper by 
Shrader-Frechette in Sec. VI.) The author is a sociologist 
and his theme, put simplistically, is that both scientists 
and philosophers practice within social groups, that their 
practice differs, and that these differing practices affect 
epistemological or analytical approaches. In order to 
"influence others' actions," applied philosophers "might 
have to engage actively in the social life of their subjects 
[scientists] .. .instead of relying almost exclusively on 
analysis of the general structure of scientsts' intellectual 
accomplishsnents. " 

Mulkay, M.J., "Three models of scientific development," The 
Sociological Review . 23 (August 1975); 509-526. File "Sociology 
of Science." 

This is a historically-oriented sociological attempt to 
construct three alternative models of scientific 
development: models of "openness," "closure," and 
"branching," With a reply by John Parker. 

Reif , Fred and Strauss, Ansela, "The impact of rapid discovery 
upon the scientist's career," Social Problems . 22 (Winter 1965): 
297-311. File "Measuring Output." 

Account of successful scientists* career paths. 

Singer, Barry F. , "Toward a Psychology of Science," American 
Psychologist . 23, (Nov. 1971) : 1010-1015. File "Sociology of 
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Science. " 

Builds on S.S. Stevens' early vrork on "science of science." 
Attempt to blend "psychology, science, and philosophy of 
science." 

Zuckeraan, Harriet, "Theory choice and problem choice in 
science," Sociological Inquiry . 48 (1978): 65-95. File 
"Sociology of Science." 

Reviews "current sociological and philosophical work bearing 
upon cognitive change [in science]." Finds that scientists' 
behavior corresponds but little to "epistemological 
prescriptions of how they should behave," but "corresponds 
more than one might suppose to often stated methodological 
precepts which give primacy to assessed significance and 
feasibility of solution." 



X. Methodological papers and blblicsraphies fsee the intro duction 
for thg more important bibliographies) 

Glaser, Edward M. and Blacker, Thomas E., "Outline of questions 
for program evaluators utilizing the clinical approach," 
Evaluation , i (Fall 1972): 56-60. File "Disciplinary." 

Basic summary of policy evaluation, one short section on 
"Methods of Measurement," discussing appropriateness of 
"objective" or "subjective" techniques. 

Guttman, Louis, "A basis for scaling qualitative data," American 
Sociological Review . £ (April 1944): 139-150. File "Bibliography 
and Methodology." 

Source of the "Guttman Scale." 

Hahn, Roger, A Bibliography of Quantita tive Studies on science 
and its History (Berkeley, Calif.: University of California at 
Berkeley, Berkeley Papers in History of science, no. 3, 1980). 
Shfclf . 

Kenwood, Felicity and Thomas, Graham, Science. Teehnolocrv. and 
Innovation: A Research Bibliography (Brighton, U.K.: Wheatsheaf 
Books/Harvester Press, 1984), Chapter 1, "Measurement of science 
and technology." File "Bibliography and Methodology." 

Lists primarily, but not exclusively, statistical 
publications of governments and international organizations; 
all entries are held by the Science Policy Research Unit of 
the University of Sussex. 

Holton, Gerald and Sopka, Katherine, "Great books of science in 
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the twentieth century: Physics," in Great Ideas Todav! 197? 
(Chicago: Encyclopedia Britannica, 1979): 225-277* Pile 
"G* Holton Papers 

McFadden, Daniel, "Conditional logit analysis of qualitative 
choice behavior," Chap. 4 in P. Zaremska, ed.. Frontiers in 
Econometrics (N.Y.: Academic Press, 1974): 105-142. "Bibliography 
and Methodology." 

Only the introduction and conclusion are filed. 

Social Science Research Council, Center for the Coordination of 
Research on Social Indicators, "Statistical data on science 
collected by federal agencies — h draft bibliography," prepared by 
R.B. Miller, April 22, 1977. File "Bibliography and 
Methodology." 

Sussex, University of. Science Policy Research Unit, "Recent SPRU 
vork on research evaluation," 1985. File "Bibliography and 
Methodology. " 

Vetter, Betty M. and Jensen-Fisher, Susan, Guide to Data 
Scientists and Engineers (Washington, D.C. : Scientific Manpower 
Cciamission, April 1984). Shelf. 

Includes three indexes. The first is organized by publisher. 
The publishing organization (about 50 are listed) is 
described, as well as that organization's "manpower 
surveys." The second organizes publications according to 
field, the third by year of publication. 
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XI. Aecees of women to part^ clpation In aclentlflc research 

All entries are in project file "Access to Science" unless noted 
otherwise . 

For further references, see also the following papers, which have 
especially useful bibliographies. (See below for full cites of 
these two papers.) 

Berryman, Suo E. , who will Do Science? 

Cole, Jonathan R. and Zuckerman, Harriet, "The productivity 
puzzle. " 



American Association for the Advancement of Science, Office of 
Opportunities. in Science, "Associations and committees of or for 
women in science, engineering, mathematics and medicine," 
compiled by Alicia E. Leach and Michel e Aldrich (AAAS Publication 
84-6, December 1984). Shelf. 

A compilation of information on women's groups by field; 
includes the social sciences. Each entry includes contacts, 
with address and phone number, membership, publications, 
current activities, and future plans, copies can be 
obtained from AAAS, Office of Opportunities in Science, 1776 
Massachusetts Avenue, Washington, D.C. 20036. 

Astin, Helen S., "Academic scholarship and its rewards," in 
Marjorie W. Steinkamp and Martin L. Maehr, eds.. Women in Science 
(Greenwich, Conn.: JAI Press, 1984): 259-279. Shelf. 

Astin states in her abstract that "This paper challenges a 
frequent conclusion that women faculty lag behind their male 
counterparts because of their lower research output." She 
specifically contrasts the findings of her research, which 
takes into account factors such as marital status, age, 
and characteristics of graduate institution and employer 
institution, with those of J. R. Cole iu liis book, Zair 
Science (see below) . 

Astin, Helen S., "Factors affecting women's scholarly 
productivity," in Helen S. Astin and Plrmer Z. Hirsch, eds.. The 
Hiahe- ? - Prmcation of Women (New York: Traeger, 1978): 133-57. 



•This article investigates the reasons why women scientists 
produce less published research. The author identifies 
several important variables related to productivity. She 
finds that marriage is positively correlated with 
productivity . 

Astin, Helen S., The Woman noctorata in Ame rica: origins. Career^ 
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and Family (New York: Russell Sage Foundation, 1969) • Widaner 
Soc 5085.169. 

One of the first studies perfcr.rii?d on women scientists urging 
original data and rigorous socicjl fiir.^i^^'ST^if ic metJiods. h 
principal conclusion is that marriage auc. rhiXCl-r?»Aring saay 
delay to some extent, but do not qualitatively inliibit 
career achievement. Includes discussions of demographic 
characteristics, early career choices, work patterns, 
occupational rewards, family life, obstacles to professional 
advancement, and several autobiographical sketches. 

Astin, Helen S. and Bayer, Alan E., **Pervasive sex differences in 
the academic reward system: Scholarship, marriage, and what 
else?" Chapter 10 in Darrell R. Lewis and William E. Becker, Jr., 
eds,, ^^cademlc Rewards in Higher Education (Cambridge, Mass.: 
Balllnger Publishing Co., 1979): 211-29. 

The authors study sex differences with respect to rank and 
salary. They find discrimination to be a significant factor 
in determining the observed lower rank and earnings 
of women. They suggest that married women, whose earnings 
and rank are more on a par with men than single woxaen, may 
be discriminated against less because of social norms 
favoring women who have the married status. 

Astin, Helen S. and Hlrsch, Werner Z., eds.. The Higher Education 
of Women: Essavs in Honor of Rosemary Park (N.Y.: Praeger 
Publishers, 1978). Wldener L.C. LC.1567 H53. 

Includes a number of anecdotal accounts and qualitative 
analyses. See also entries under Astin and under Hlrsch and 
Hlrsch in this section. 

Bachtold, Louise H. and Werner, Emmy E., "Personality profiles of 
gifted women: Psychologists," in Eiduson, Bernice T. and Beckman, 
Linda, eds.. Science as a Career Choice (New York: Russell Sage 
Foundation, 1973): 551-63. Shelf. 

The authors find that their sample differs from "adult women 

in general and from women college students in many of the 

same personality characteristics on which they resemble 

academic men." They also discuss data on publication 

and work patterns of developmental, counseling, and clinical 

psychologists. 

Bayer, Alan £. and Astin, Helen S., "Sex differentials in the 
academic reward system, " Science ^ 188 (May 23, 1975): 796-801. 

The authors assess the progress of women scientists since 
the implementation of federal antibias regulations in the 
early 197 Os« Comparison of e)cpected (on the basis 
of productivity) rewards and actual rewards reveal 
significant differentials, which are seen to be the result 
of continuing sex bias, though this bias is thought to be 
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less than five years earlier. Astin and Bayer's 1979 paper 
(above) builds upon this one. 

Benbow, CatiiHa Persson and Stanley, Julian C. , "Gender and the 
science maj'^ri A etudy of mathematically precocious youth," in 
Marjorie W. JUtainlcMmp and Martin L. Maehr, eds.. Women in Science 
(Greenwich, Conn.: JAI Press, 1984): 165-96. Shelf. 

Studies seventh and eighth grade students with unusually 
high aptitude for math; more girls than boys among th^iiie 
students later took high school physics, and the boys scored 
higher on standardized tests in the natural sciences. The 
authors conclude only that some of the difference can be 
accounted for "by the females' lower mathematical reasoning 
ability in the seventh and eighth grade," although all of 
the mathematically precocious youth had a more enthusiastic 
attitude toward science. 

Berryman, Sue E. , Who Will Do Science? (New York: Rockefeller 
Foundation Special Report, November 1983). Shelf. 

Comprehensive study of women and minorities based on a 
considerable body of data; includes relatively elaborate 
attempt to model and explain female and minority 
underrepresentation in the sciences. See also entry under 
Scientific Manpower Commission. 

Blackburn, Robert T. , Behymer, Charles E. , and Hall, David E. , 
"Research note: Correlates of faculty publications," Sociology of 
Educatio n. 51 (January 1978): 132-41. 

The authors survey a large group of faculty members (1,216) 
to ascertain the correlates of published productivity. They 
study faculty in both the sciences and the hiomanities and 
find some distinctive patterns in the two fields. However, 
they find that across fields professors in smaller teaching 
colleges publish less than their counterparts in large 
research-oriented universities. They also find that "Those 
who will be productive over their full careers are the 
individuals who start early, receive their degrees when 
young, and take on the habit of regular output." The authors 
find that "sex is not a significant predictor" of 
productivity. They go on tc, . - g'^est in a footnote, however, 
that discrimination my account :n part for women's lower 
productivity. 

Bruer, John T., "Women in science: Toward equitable 
participation," Science. Technol ogy, s Human Values. S. (Summer 
1984): 3-7. WOM. 

A brief review of recent Macy Foundation activities with 
respect to women in science, including an account of 
quantitative and qualitative research by J.R. Cole and H. 
Zuckerman . 
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Brush, Stephen G. , "Women in physical science: From drudges to 
discoverers, w The Physics Teacher (January 1985): 11-19. 

A short essay including biographic portraits of 10 
outstanding women scientists. 

Carnegie Institution of Washington, Jeunes D. Ebert, President, 
"Report of the President: 1982-1983." Shelf. 

Includes a section on the work of Barbara HcClintock, who 
received support from the Carnegie Foundation and won the 
Nobel ??ri2e during the year covered by this report. 

Centra, John A., Women > Men and the Doctorate (Princeton, N.J.: 
Educational Testing Service, 1974). Widener Soc 5085.174.25. 

This is a large-scale study of women with doctorates based 
on a questionnaire survey (sample sizu: 3,068 men and 
women) . The researchers gather data on careers of women and 
men: promotion history, interests in teaching and research , 
marital and family status, publication rates, etc. Subjects 
were also asked about their graduate school c^eriences and 
their attitudes toward women *s rights. Centra concludes by 
defining four types of female? career paths, the distinctions 
between which are largely determined by cultural tradition, 
BBK role expectations, and discriminatory practices. 

Chvibin, Daryl, "Sociological manpower and womanpower: Sex 
differences in career patterns of two cohorts of American 
doctorate sociologists," The American Sociologist ^ £ (May 1974): 
83-92. C02ZENS. 

Studies demographics, research performance, and mobility of 
men and women. Finds that "the structure of [sociological] 
employment militated against women sociologists receiving 
recognition by discouraging their mobility to the 
prestigious Ph.D. granting sector. At the same time, the 
vehicle for attaining a job in this visible 
sector — published research — was underutilized." Chubin 
suggests that this was due to subtle discrimination in 
graduate school and in the profession. 

Cole, Jonathan R. , Fair Science: Women in the Scientific 
Community (New York: The Free Press, 1979). Kennedy School 
Library Q.130 C64. 

A study of sex discrimination in science. (The book does' 
not try to determine the causes of women's lower 
productivity • ) Cole concludes that discrimination against 
academic women scientists is probably a significant factor 
only with respect to promotion to the tenured faculty ranks. 
(See also the review by Roark, listed below.) 

Cole, Jonathan H. and Cole, Stephen, Social stratification In 
Science (Chicago: University of Chicago Press, 1973). Cabot 
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Library HM.38 C65. 

Includes a chapter on women and minorities in science, and 
possible discrimination against them. 

Cole, Jonathan R. and Cole, Stephen, "The reward system of the 
social sciences," in Charles Frankel, ed. , controversies and 
Decisions , f \<Iew York: Russell Sage Foundation, 1976): 55-88. 

Tha Coles investigate the extent to which scientists are 
discriminated against in academic settings on the basis of 
uneir social status, including sex. In this early 
article, the authors state their inability to explain 
sex-based productivity differentials and find that 
discrimination is not a factor in the granting of financial 
support for women and men's graduate study. 

Cole, Jonathan R. and Zuckerman, Harriet, "The productivity 
puzzle: persistence and change in patterns of publication of men 
and women scientists," in M.W. Steinkamp and M.L. Maehr, eds., 
women in Science (Greenwich, Conn.: JAI Press, 1984): 217-58. 
Shelf. 

An important piece of research on productivity differentials 
between men and women scientists. However, the authors do 
not explain productivity differences; they only show that 
they have persisted over the last several decades with very 
little change in magnitude. They offer suggestions for 
further research. 

Diamond, Diana, "Study shows grad women have less 
self-confidence, despite high marks," Campus Report (Stanford 
University), Nov. 14, 1984. 

Summarizes a study conducted by Larraine Zappert of 
Stanford. 

Eccles (Parsons) , Jacquelynne, "Sex differences in mathematics 
participation," in Marjorie W. Steinkamp and Martin L. Maehr, 
eds., women in Science (Greenwich, Conn.: JAI Press, 1934): 
93-137. Shelf. 

An extended analysis of the factors contributing to girls' 
choosing to study math less frequently than boys; attempts 
to place the data within the context of psychological 
theory. 

Epstein, Julia L. , Review of Vivian Gornick, Women jn Scjence; 
portraits from a World in Transition , in Isi.s, 75 (1984): 578-79. 

"The strength of women in science emerges from the indivdual 
voices of those Gor- _ick interviews. . .Unfortunately, her 
impressions float -.-ee of historical analysis. . .ultimately 
the book never fully supports its thesis [that the practice 
of contemporary scienr" harbors a key to the recent 
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sociology of feminism] . " 

Berber, Marianne and Hxaber, Joan, "Husbands, wives, and careers," 
Journal of Marriage and the Family . Al (May 1979): 315-25. 

The authors attempt to determine the extent to which a 
spouse's level of education affects the careers of Ph.D. 
recipients. They have e sample of 346 men and 576 women, 
employed in academia« industary, and elsewhere, who are 
assigned to four disciplinary categories: physical , 
biological, and social sciences, and the humanities. They 
fovind that for both men and women, marriage to a highly 
educated spouse is associated with a fertility decline; 
marriage to a Ph.D. adversely affects a female Ph.D.'s 
labor-force participation, apparently because of constraints 
placed upon her due to her hub^und^s need for mobility; and 
marriage to a Ph.D. adversely affects a male Ph.D. 's 
published productivity and/or offices held. 

Ferber, i^Iarianne A. and Loeb, Jane W., "Performance, rewards^ and 
perceptions of sex discrimination among male and female faculty," 
American Journal of Sociology , 78 (January 1973): 995-1002. 

The authors use bivariate correlations in this study of 
women and men faculty members (scientists and others) at the 
University of Illionois, Champaign- Urbana. They found that 
marriage and children did not affect the productivity of 
women faculty members (where productivity is operational ized 
as publications of various kinds) • Marital and parental 
status is correlated with higher salaries for men, possibly 
because of perceived financial need on the part of 
extipl oyer s. Perception of discrimination seemed to be more 
accurate among women than mcri (i.e. women had more awareness 
of "salary and rank inequities") . 

Florman, Samuel C, "Will women engineers make a difference?" 
Technolocrv Review . 87 (November/December 1984) : 51-52. Under 
Tech. Rev, in file. 

An essay on women in engineering, suggesting that 
because they "bring a new dimension to the prof ession. . . a 
more philosophical and aesthetic concern," and because of 
their broader view and better comaiunication skills, it is in 
society's interest to encourage women to become engineers. 

Fox, Lynn H. , The Problem of Women and Mathem atics! A Report to 
the Ford Foundation (New York; Ford Foundation, March 1980) . In 
project file (not on project shelf.) 

Discusses sex differences in mathematics, influence of 
teachers, counselors, and parents, factors affecting career 
choice, suggestions for future research, and recommendations 
with respect to educational policy. 

Frank, Robert H. , "Family location constraints and the geographic 
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distribution of female professionals," Journal of Political 
Economy p 86 (February 1978): 117-30. 

Essentially a variation of Frank's article in American 
Economic Review (see below) • Discusses implications for 
public policy. 

Frank, Robert , "Why women earn lees: The theory and estimation 
of differential overqualif icatiou," American Economic Review . 68 
(June 1978); 360-373. Widener. 

This is a mathematical formalization of the job location 
process for two-career married couples. The author analyzes 
the consequences of the tendency of married couples to 
maximize the husband's professsional potential by choosing 
to locate themselves in geographic region in which he has 
found his bBst. job. . This results in the differential 
overgualif ication of professional women. See also the 
article by Harwell, et al. and by Rosenfeld (1981), and book 
edited by Pepitone-Rockwell (all in this section) . 

Frieze, Irene Hanson and Hanusa, Barbara Hartiuan, "Women 
scientists: Overcoming barriers,'" in Marjorie W. Steinkamp and 
Martin L. Maehr, eds. , Women in Science (Greenwich, Conn.: JAI 
Press, 1984): 139-63. Shelf. 

Studies barriers which women confront ^ from secondary school 
through professional work, with respect to careers in 
science; includes a discussion of possible factors involved 
with women scientists'* lower productivity such as family 
obligations and differential access to informal social 
networks . 

Garfield, Eugene, "The 1983 Nobel Prizes. Part 2. Myth or 
reality: Premature discovery is not the same as being ignored 1 
Barbara McClintock and the Prize in Medicine," Current , Contents 
(Feb. 18, 1985) : 3-10. 

Critiques Evelyn Fox Keller's biography of McClintock. (See 
below. ) 

Garfield, Eugene, "The 1986 NAS Award for Excellence in 
Scientific Reviewing goes to Virginia L. Trimble for her reviews 
in astronomy and astrophysics," Current Contents (May 12, 1986): 
4-11. 

Discusses Trimble ^s work in citation analysis as well as her 
research and reviewing in astronomy and astrophysics. 

Garfield, Eugene, "Why aren't there more women in science?" 
current Contents (April 26, 1982): 5-12. 

A brief account of some recent research in the field, with a 
list of the 27 most-cited female scientists for the period 
1965-1978, vith their affiliations. 
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Graham, Patricia Albjerg, "Women in academe," Scile:v 16£ 
(September 25, 1970): 1284-90* 

A review of some of the problems faced by fe^r^ academics 
in the late 1960s; includes a historical acc *^ of women in 
universitios in the U*S, 

Grieb, Aimee and Easley, Jack, "A primary school impediment to 
mathematical equity? Case-studies in rule-dependent 
socialization," in Marjorie W. Steinkamp and Martin L. Maehr, 
eds., women in Scien ce (Greenwich, Conn.s JAI Press, 1984): 
317-362. Shelf. 

Through case studies of elementary school students, the 
authors attempt to "identify a social mechanism in primary 
schools which allows white, middle-class male students who 
are creative in their study of mathematics to preserve an 
independent attitude while keeping . females and minorities in 
an attitude of dependence on knowing the algorithm before 
proceeding with the problem." 

Haas, Violet B. and Perrucci, Carolyn C. , eds. Women in 
Scientific and Engineering Professions (Ann Arbor, Mich.: 
University of Michigan Press, 1984). Shelf. 

A collection of papers presented at the Conference on Women 
in the Professions: Science^, Social Science, Engineering, 
held at Purdue University in March 1981. The piece is by 
Rosenfeld is included below. 

Hansen, W. I/-^ Weisbrod, Burton A. and Strauss, Robert P., 
"Conf irjaatio ; and contradictions: Modeling the earnings and 
research productivity of academic economists Journal of 
Political Econow . 86 (October 1978): 729-41. 

The paper does not focus on rafj^le--female differentials, but 
is relevant to the issue as it i:>rovides a useful economic 
analysis of earnings. The authors conclude that published 
productivity has a much gsreater effect on earnings than 
previous studies have suggested. 

Hargens, Lowell L. , McCann, James C. , and Reskin, Barbara F. , 
"Productivity and reproductivity: Fertility and professional 
achievement among research scientists," Social Forces . 52 
(September 1978): 154-63. 

Directly addresses the question of whether having children 
affects the piablished productivity of a group of research 
chemists,* the authors find that both women and men display 
decreased productivity after having children, to 
approximately the same degree. Note that Reskin ( American 
Journal of SociologY p 1978) apparently uses approximately 
the same data set but finds no relationship between family 
responsibilities and productivity, either negative or 
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positive. 

Harnisch, Delwyn L. , "Females and aathematicsi A cross-national 
perspective," in Marjorie W. Steinkamp and Martin L. Maehr, eds., 
Women in Science (Greenwich, Conn.: JAI Press, 1984): 73-91. 
Shelf. 

Analyzes data on sex-differences in mathematics for women in 
Japan, Sweden, and the United States. 

Harway, Michele and Astin, Helen S., Sex Discriininatio n in Career 
Counseling and Education (N.Y.i Praeger, 1977). Widener L.C. 
I1BIO27.5 .H355 1977. 

Comprehensive and very useful account of the early 
socialization process of female high school students and of 
sex bias on the part of counselors; does not focus on 
science in particular, but see entries for "mathematical 
ability" and "mathematics and science: importance of high 
school preparation in" in the index. 

Helmreich, Robert L. , Spence, Janet T. , et al., "Making it in 
academic psychology: Demographic and personality correlates of 
attainment," Journal of Personality and Social Psvcholoov . 39 
(1980): 896-908. 

An important article, referenced favorably by Colii and 
Zuckerman in their recent work (see above), whir-b iiiciu ' js 
data on sex differences among academic psycholo.Tfi rsts., Tho 
authors had a sample of 141 male and 55 female 
psychologists, to whom they administered two m a k<. 
relevant personality variables, and for whom ta^-X' collected 
publication, citation, and de5^i;.:>riir.r. ;c data. They develop a 
relatively complex causal aodfe - attainment in psychology 
on the basis of their data in :^riv.>;-.' sex, quality of graduate 
school, and quality of curreji^. are causal to 

attainment. In addition, cert::" motivational atrributes 
have an effect on productivity attainment. The usual 
sex rilfferences in productivity were observed. Marital and 
fati^ responsibilities were not seen to affect 
productivity. Measured personality differences did not 
account for productivity differences, either. The authors 
speculate that social -psychological factors which they did 
not study may be important in this regard, tsuch as isolation 
of women from informal collegial networks, and citation 
biases on the part of piablishing psychologists. 
Interestingly, the authors suggest that "The substantial 
differences between men and women in attainment measures, 
particularly in citations, may. . .iiepre sent the aggregate 
effects of a large number of relatively independent and/or 
interactive causes, each quite minor when considered by 
itself." 

Helson, Ravenna, "Women mathematj.cians and the creative 
personality," in Eiduson, Bernice T. and Beckman, Linda, eds.. 
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Science as a Career Choice (New York: Russell Sage Foundation, 
1973): 563-74. Shelf. 

The authors compare creative women mathematicians with other 
women holding Ph.D.'s in mathematics. Their sample includes 
45 women, of vbom the 18 '^creative'* individuals are 
"virtually all of the creative women mathematicians in the 
United States.*' The authors identify seve^ral personality 
traits characteristic of the creative io.^thematicians, but do 
not believe that they have different ^orts of cognitive 
abilities than the other women in the sample. A central 
question in the study is "how a woman could so suppress her 
feminine nature to be a mathematician without suppressing 
her originality also." They tentatively answer that women 
mathematicians are introverts but, more importantly,- the 
creative women are able to express themselves freely in 
their work, "with emotional involvement." 

Hirsch, Hilde E. and Hirsch, Werner Z., "Intellectual quality: 
The symbols and the svibstance," in Helen S. Aetin and Werner Z. 
Hlrsch, eds., The Higher Education of Women (New York: Praeger, 
1978) : 161-65. 

An essay on intellectual quality; emphasizes in particular 
the inadequacy of quantitative measures, especially citation 
counts, as a measure of academic ability, and suggests that 
"women and minorities are likely to suffer from a reliance 
on surrogates [indirect, quantitative measures] that often 
poorly reflect true ability." 

Hoffleit, Dorrit, "Confirmed stereotypes transforiaed to freedom 
and variety," review of Vivian Gornick, Women i w S cience; 
Portraits from a World in Transition , in Physic cv Today (September 
1984) : 77-78. 

Largely positive; quotes Gornick describinoi her book as 
"•impressionistic journalism. '" 

Homig, Lilli S. , "Women in science and engineering: Why so few?" 
Technolocfv Review , 87 (November/December 1984): 30-41. Urier 
" Tech. Rev. " in file. 

A historical overview of discrimination against women in 
science and engineering. 

Humphreys, Lloyd G. , "Women with doctorates in science and 
engineering," in Karjorie W. Steinkamp and Martin L. Maehr, eds., 
Women_in Science (Greenwich, Conn.: JAI Press, 1984): 197-216. 
Shelf. 

A detailed study of the characteristics of female Ph.D. 
degree holders and the problems which they have encountered; 
concludes that men and women "are essentially equal on the 
general factor in intelligence," and that sex-related 
differences in certain intellectual skills are 
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environmentally determined. 

Johnson, George Ht. aiid Stafford, Frank P., "Pecuniary rewards to 
men and women faculty," Chapter 11 in Darrell R. Lewis and 
William E. Becker, Jr. r eds., Aeademic Rewards in Higher 
Education (Cambridge, Mass.: Ballinger Publishing Co., 1979): 
231-243. 

This article is an attempt to explain salary differences 
between men and women faculty members. The authors explore 
two alternative exlanations for women's lower salaries: 1/ 
women choose to aOJsent themselves from the labor parket, 
primarily for family reasons, produce less, and thus are 
paid less? and 2/ women are discriminated against. They 
find evidence for both, and explore the policy implications 
which follow from their analysis. 

Johnson, George E. and Stafford, Frank P., "Women and the 
academic labor market," Chapter 8 in Cynthia B. Lleyd, ed. , sex. 
Discrimination, and the Division of Labor (New York: Columbia 
jniversity Press, 1975): 201-19. 

This earlier article of Johnson and Stafford's is the basis 
for their 1979 chapter. (See above.) Importantly, 
they find that "women who are married are most likely to 
drop out of full-time employment, particularly when 
preKchool-age children are present in the household." He 
cites Astin's similar findings in The Woman Doctor ate in 
Ajflerica . 

Kaufman, Debra Renee, "Associational ties in academe: Some male 
and female differences," sex Roles . 4. (February 1978): 9-21. 

An article on social psychological factors involved with 
acdemic productivity. Kaufman concludes that her "analysis 
of the collegial-friend relationships among academicians... 
indicates that female professors, especially unmarried, have 
fewer males in their collegial-friend networks than men. 
Whether by choice or exclusion, it is suggested that 
isolation from these informal collegial contacts leaves 
women at a profound professional disadvantage." Kaufman 
does not indicate the disciplinary affiliation of her 
subjects. They may not be scientists. 

Kavrell, Suzanne McNeill and Peterson, Anne C. , "Patterns of 
achievement in early adolescence," in Marjorie W. Steinkamp and 
Mar+-^n L. Maehr, eds.. Women in Science (Greenwich, Conn.: JAI 
1984): 1-35. Shelf. 

Studiocj the possible roots of sex-related differences with 
rerpect to science; suggests that during the period between 
slxt:c» and eighth grades, girls with aptitude for science may 
require special encouragement. 

Keller, Evelyn Fox, "Contending with the masculine bias in the 
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ideals and values of science," The Chronicle of Hlahor Education 
(Oct. 2, 1985) : 96. 

An essay on traditionally feminine xaodes of i;iguiry: feeling, 
identification with nature, love. Argues that these are 
vaXueible in the pursuit of knowledge. 

Keller, Evelyn Fox, A Feeling for the Qrcranism! The Life and Work 
of Barbara McClintock (San Francisco: W.H. Freeman, 1983). Cabot 
Library QH.429.2 M38 K44 1983. 

A biography of the Nobel Prize winning biologist, focusing 
in part on what Keller believes to be the special cognitive 
abilities women bring to science. See also the Technology 
Review article by Keller below and the review by Bentley 
Glass in Ists , 75 (September 1984): 600-601. 

Keller, Evelyn Fox, Reflections on Gender in Science (New Haven, 
Conn.: Yale University Press, 1985). 

Investigates the perceptual and epistemological dimensions 
of the subjective, intuitive, and emotive approach to 
gaining knowledge usually characterized as feminine. 

Keller, Evelyn Fox, "Women and basic research: Respecting the 
unexpected," Technology Review ^ 87 (November/December 1984): 
44-47. Under Tech^ JReVo in file. 

Studies possible differencer» between men and women in 
epistemological approaches to science; focuses on the work 
of Barbara McClintock. 

Kevles, Daniel J., "Statistical data and the history of women: A 
critique of Margaret Rossiter's Women Scientists in America: 
Struqcrles and Strategies to 1940 ." California Institute of 
Technolog\% Division of the Humanities and Social Sciences, 
Humanities Working Paper No. 79, March 1983. 

Author- s abstract: "Rossiter*s book. ..is a goldmine of 
information. At its cere is a statistical data base drawn 
from succesive editions of American, Men in Science . . . 
However, Rossiter makes no standard tests of significance of 
her valuable statistics. More important, she commits the 
major methodological sirii of giving inadequate attention to 
alternative explanations of the numerical data. The result 
is that while Rossiter amply documents the considerable 
discrimination that women faced in the American scientific 
enterprise, she leaves cloudy the relative force of that 
discrimination compared to internalized cultural norms, 
marital and naternal obligations, and the like." 

Kolbert, Elizabeth.^ "Scientific ideas: Women's vs. Men's," New 
York Times p Oct. 17, 1985^ pp. cl, cl2. 

Review of the work of Ruth Bleier, Ruth Perry, and Evelyn 
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Fox Keller, who in various ways, on the basis of their 
research, believe that nien and women have different 
cognitive approaches to science. 

Kremer, Barbara K. , ••The meta-analysis of gander differences in 
science learnings A first step toward the development of 
educational policy to exisourage women in science," in Marjorie W, 
Steinkamp and Martin L. Maehr, eds.. Women in Science (Greenwich, 
Conn.: JAI Press, 1984): 51-71. Shelf. 

Reviews and "meta-analyzes" research on sex-related 
differences in science and on science education; concludes 
that carefully planned educational intervention to encourage 
women's interest in science can be effective. 

Long, J. Scott, "Productivity and academic position in the 
scientific career," American Sociological Review , ^3 (December 
1978) : 889-908* 

Long's paper is interesting for the extreme position that it 
takes. He does not examine women in particular, but rather 
looks at the relationship between productivity and position 
in the '^i^ademic stratification. system. " His subjects are 
biochemists. He concludes from his research that position 
within the academic system determines productivity more 
strongly than vice-versa. He suggests that this is true 
from the very beginning of the academic career. 
Particularistic, rather than universalistic, factors 
determine productivity at all points in the academic career. 
He (explicitly argues against Cole and Cole who, in Social 
stratification in Science , suggest that accumulative 
advantage partially accounts for stratification, but that 
productivity is the dominant factor in determining rewards 
in the beginning of a scientific career. 

Lotze, Barbara, ed., jiakina Cortributlons: An Historical overview 
of Women's Role in Fhysjcs (College Park;. Maryland: American 
Association of Phyj^ics Teachers^ 1984). Shelf. 

Historical ciocounts of prominent women in physics. 

Martin, Ben R. and Irvine, John, "Women in science — the 
astronomicai brain drain," Women's Studies International Forum , 
5, No. 1: 41-68. In File "Martin/Irvine." WOM. 

The authors wrote this article as part of a long-term study 
on the evaluation of large-scal^^ basic research in Great 
Britain. The subjects are all i'h.D. students in radio 
astronomy in Great Britain, from the emergence of the 
discipline in 1947 through 1977 (14 women and 301 men). 
There is an observed discrepancy between the performance of 
the women in graduc\te school, which was excellent on 
average, and their lack of siibsequent achievement, relative 
to the men. The authors conclude that "the lack of relative 
success. ..is most likely to be the result of social factors 
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concerning the role vlthin the family rather than any 
inherent lack of ability as a scientist or direct 
discrimination in employment." Specifically, it was found 
that disproportionate nuiabers of women radio astronomers 
dropped out of the scientific work force upon marrying or 
having children. 

Harwell, Gerald, et al., "Geographic constraints on women's 
careers in academia," Science , ?05 (September 21, 1979): ir. ''^-31. 
Cabot Library. 

The authors suggest that when both members of a married 
couple are seeking professional positions, the couple is 
more likely to choose the geographic location in which the 
husband's earnings and potential for career advancement aro 
maximized. Using data they have collected and analyzed, 
they state that female academics are concentrated In large 
urban areas. They suggest that this location pattern is due 
to professional academic wives being able to find a job when 
their husbands gain academic or other employment in a large 
city (because there are more jobs available), but being less 
able to find jobs when their husbands gain employment in 
universities located in smaller towns. See also articles by 
?ranlc and by Rosenf eld (1981) , and book edited by 
Pepitone-Roclcwell . 

National Academy of Sciences, Cliytblnq the Academic Ladder: 
Doctoral Women Scientists in Academe (Washington, D.C. : National 
Academy Press, 1979). Cabot Library Q.130 N37. 

A study by the Committee on the Education and Employment of 
Women in Science and Engineering of the National Research 
Council on the progress of women In academia during the 
mid-1970s and their continuing problems with discrimination 
and, to a lesser extent, other problems leading to 
imbalanced hiring and promotion practices. 

National Academy of Sciences, Climbing the Ladder: Ar ' ^ ^ on 
the statiiia of Dochoral Women Scientists and Engineers 
(Washington, D.C: National Academy Press, 1983). Cabot Library 
Q.130 C54 1983. 

An update of the 1979 study by the National Research 
Council, including data on women scientists and engineers in 
industry. 

National Science Foundation, Women and Minorities in Science and 
Engineeerlng (Washington, D.C: DSGPO, Jan. 1977, Jan. 1982, Jan. 
1984, Jan. 1986). Shelf. 

Data on employc^ent levels and trends, fields of employment, 
experience, career patterns, and scientific and technical 
education. 

Pepitone-Rockwell, Fran, ed., Dual-Career Couples (Beverly Hills, 
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Calif.: Sage Publications, 1980). Littauer Library HQ. 536 D797. 

Compilation of research on the sociology and economics of 
dual-career couples three sections on 1/ historical 
development of dual-career couples, 2/ marriage and family 
issues, and 3/ career issues. See also articles by Harwell 
and by Prank. 

Reskin; Barbara F., "Sex differences in status attainment in 
science: The case of the post-doctoral fellowship," American 
Sociological Review . Al (August 1976): 597-612. 

Reskln's principle finding is that for men, the nxpected 
relationship between the calibre of doctoral trcxining and 
graduate school performance, and the p'*estige of the 
postdoctoral award, was evident. For women, this 
relationship was not evident. Two explanations are 
considered: gender differences in professional commitment 
and discrimination. Reskin finds evidence for both 
explanations and finds that they are not mutually exclusive 
because "regardless of their level of professional 
commitment, female scientists as a group also may be 
somewhat more commited to extraprofessional (primarily 
familial) roles than are men." 

Reskin, Barbara F. , "Sex differentiation and the social 
organization of science," Sociological Inquiry . 18, No. 3-4 
(1978): 6-37; reprinted in Jerry Gaston, p-d.^ The Sociologv of 
Science (San Francisco: Jossey-Bass, 1978): 6-37. 

This article is a literature review and theoretical 
investigation of women's role in science. Reskin offers no 
original field research, but does offer some original 
anecdotal evidence. Her principal argument is that because 
women's status within the scientific profession is in 
general lower than that of men (for reasons discussed in the 
paper) , men and women are likely not to interact as full 
colleagues. Women then may focus on technical duties, 
consistent with their lower status, and not perform as much 
original research. See also articles by Widom and Burke and 
by Kaufman >all in this section) . 

Reskin, Barbara F., "Scientific productivity, sex, and location 
in the institution of science," American Journal of Sociology, fil 
(March 1978): 1235-43. 

Reskin looks at female and male chemists. She attempts to 
identify determinants of scientific productivity 
(publication) and how thtcc determinants differ for men and 
women. She then notes that organizational features of 
science, rather than gender, which l3 functionally 
irrelevant for scientific p2rfonri-.::ce, might be expected to 
account for differences in the determinants of productivity 
and in the productivity itself. 
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Reskin^ Barabara P. ^ and Hargens^ Lowell L.^ ^scientific 
advancement of male and female chemist:.;," Chapter 5 in Rodolfo 
Alvarez and Kenneth 6. Lutterman, ede.; Otscrimination in 
Orqaniyations; Using Social Indicators to Manage Social Chw e 
(San Francisco: Jossey-Bass Publishers, 1979): 100-22 r 

This paper is based directly on the 1978 article in the 
A merican Journal of Sociology , (See above.) However, the 
authors discuss discrlsxination more explicitly as a factor 
which determines women's position In the organizational 
structure of science. 

Richter, Derek, ed.. Women Scientists: The Road to Liberation 
(.London: Macmillan, 1982). Cabot Library Q.130 W67 1982. 

A fascinating series of autobiographical accounts, written 
by eminent women scientists from the following countries: 
India, U.S., Japan, France, Italy, Iran, Sweden, USSR, U.K., 
and Kenya. Provides insight into the comparative sociology 
of science, the conditions under which women work in other 
countries, and the lives of persistent, adventurous, and 
successful individuals. The papers were compiled partly in 
collaboration with the World Health Organization. 

Roark, Anne, "Women scientists faring better, study finds," 
Science and Government Reijort (December 1, 1979) . 

A review of Jonathan Cole's book. Fair Science; Women in the 
Scientific Communitv (see this section) , and a good summary 
of the major findings. Claims that Cole did not go far 
enough in advocating affirmative action programs and 
highlighting the problems of women scientists. 

Rosenfeld, Rachel A. , "Academic career mobility for women and men 
psychologists," in Violet B. Haas and Carolyn C. Perrucci, eds.. 
Women in Scientific and Engineering Professions (Ann Arbor, 
Mich.: University of Michigan Press, 1984): 89-127. Shelf. 

Elaboration of article in Social Science Research (see 
below) . 

Rosenfeld, Rachel A., "Acadr^mic men and women's career mobility," 
Social Science Research , IQ (December 1981): 337-63. 

* 

A study of psychologists' career paths; uses statistical 
models based in part on Markov processes to analvze 
employment transitions. The principal conclusion is that 
women are less likely to follow regiilar career paths than 
men. More specifically, Rosenfeld finds that geographic 
mobility for men seemed to depend on the nature of the job 
they were leaving, while geographic mobility for women 
"seemed to be associated only with the size of the labor 
market they were in." (See also articles by Frank and by 
Marwell, et al.) 
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Rossiter, Margaret W., "Women scientists in America before 192 o," 
American Scientist . §1: 312-323. WOM. 

The career patterns of 504 women scientists are compared 
with a control group of men. Ihe women were more educated, 
were employed more often in academic institutions, were half 
as eminent, and were more likely to be unemployed. 

Rossiter, Margaret W., Women Sci entists in America: Strucfgles- an.d 
strategies to 1940 (Bcltimore: Johns Hopkins University Press, 
1982). Widener L.C. Q130 .R68. 

Rossiter uses archival material and other sources to study 
the history of women in science from whe mid-1800 's to about 
1940. She documents discrimination aga.*' st women at each 
stage of the scientific career. AccorSiJjg to Barbara 
Sicherman, "One of Rossiter 's principal findings is the 
prevalence of sex-aggregated and sex-typed work [during the 
period under study] . . . ' See also, the review symposium in 
Isis . 7S (March 1984) : 189-203 and the revievr article by 
Kevles (listed in this bibliography. Sec. XI). 

Rubin-Rabson, Grace, "Factors in the achievement drive of gifted 
women," American Psvcholocrist . 21 (February 1971): 205-7. 

A research note, in part stimmarizing other studies on 
motivation, suggesting that during high school and college 
years, biological and personal factors are partly 
responsible for girls lack of motivation and achievement, 
and that discrimination has been overestimated as a cause of 
women's problems in higher education and in the professions. 
I'he author further suggests that legislation cannot 
alleviate the biological component of young women's 
motivational lack. 

Scientific Manpower Commission, "The science and engineering 
talent pool: Proceedings of the 1984 joint meeting of the 
Scientific Manpower Commission and the Engineering Manpower 
Commission," May 15, 1984. Shelf. 

Includes a presentation by Sue Berryman entitled "The 
"Underrepresentation of women and minorities in quantitative 
fields: How does it occurr?" 

Steinkamp, Marjorie W. , "Motivational styles as a mediator of 
adult achievement in science," in Steinkamp, Marjorie W. and 
Maehr, Martin L. , eds.. Women in Science (Greenwich, Conn.: JAI 
Press, 1984): 281-316. Shelf 

Explores possible sources of differences in mathematical 
ability between men and women; on the motivational styles of 
young boys and girls; finds that boys' motivational styles 
are "consonant" with those which are usually seen in 
scientists and girls', for the most part, are "dissonant"; 
sViCcessful women scientists display motivational 
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characteristics similar to those more generally 
representative of men. 

Steinkamp^ Harjorie W. and Maehr, Martin h. , Bds., Women in 
Science , Vol. 2 of Advances in Motivation and Achievement 
(Greenwich^ Conn.: JAI Press^ 1984). Shelf. 

A compilation of very recent work on women in science. Many 
papers in the volxime focus on the early education of boys 
and girls in science and mathematics. The papers most 
directly relevant to the problem of female scientists" 
productivity are by Cole and Zuckerman, and by Astin. See 
entries under these authors and under Kavrell and Petersen, 
Zerega and Walberg, Kremer, Hamisch, Eccles, Frieze and 
Hanusa, Benbow and Stanley, Humphreys, Steinkamp, and Grieb 
and Easley. 

TechnolocfV Review , 87 (November/December 1984): 29-52; special 
section entitled "Women in technology." In file under " Tech. 
Rev. " 

See also references to Homig, Traveek, Keller, Turkle, and 
Florman (this section) . 

Thompson, Julia, "Summary of the tenure committee survey," CSWP 
Gazette (A newsletter of the Committee on the Status of Women in 
Physics of the American Physical Society) , 4 (September 1984): 
1-2. 

Summary of study by Frieze, Thompson, and Baranger; see 
xinder Frieze (this section) . 

Traweek, Sharon, "High-energy physics: A male preserve," 
TechnolocfV Review . 87 (November/December 1984): 42-43. Under 
" Tech. Rev. " in file. 

A discussion of some of the personality traits common among 
high-energy physicists, and the difficulty with which women 
work in a community characterized by a dominance of such 
traits. This article is based on a book by Traweek entitled 
Buying Time and Taking Space: The Culture of the 
Particle-Phvsics Community . 

Turkle, Sherry, "Women and computer programming: A different 
approach," Technoloov Review . 87 (November/December 1984): 48-50. 
Under " Tech. Rev. " in file. 

Discusses differences between boys and girls in approaching 
computer programming; suggests that girls' less analytic, 
more interactive style of programming has great value and 
should be respected* 

Turner, Judith Axler, "Why more women than men shun computers," 
The Chronicle of Higher Education (October 10, 1984): 26. 
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Account of study that found women's aptitude 

for using computers as high as men, but their interest in 

doing so lower. 

Tyler, Leona E. , "Development of 'scientist' patterns of interest 
in boys," in Eiduson, Bemice T. and Bectanan, Lxnda, eds., 
Science as f Gareer Choice (New York: Russell Sage Foundation, 
1973): 285-292. Shelf. 

Studies the development and personality correlates of boys 
designated as 'scientists' or 'nonscientists' on the basis 
of the Strong Vocational Interest Blank? does not compare 
boys with girls. 

V S Congress, Office of Technology Assessment, P^Tnogy^phic 
Trends and Scien t ific and Engineering Workforce; A Technical 
yffemorandum (Washington, D.C.: USGPO, Dec. 1985). Shelf. 

Includes a chapter entitled "Demographics and equality of 
opportunity," which in turn includes a section on women in 
science. This section summarizes several demographic 
studies on women (including ones by Berryman and Vetter, 
some of whose publications are listed in this bibliography) . 
Topics include women's science education and "differential 
treatment ot women in the. . .workforce." 

Vetter, Betty M. , "Working women scientists and engineers," 
Science . 207 (January 4, 1980): 28-34. WOM. 

Abstract supplied by author: "About 80 percent of women 
trained in science or engineering are in the labor force, 
but many are employed outside of their fields. Most who 
withdraw from the labor force do so temporarily, and about 
half do not take a career break even when they have small 
children. Factors affecting labor force participation are 
student status, highest degree level, parental status and 
age of children, and field of degree. Employment 
opportunities are restricted in some fields, and women have 
higher unemployment rates and lower earnings than men. . . 

Welch, Michael R. and Lewis, Stephen, "A mid-decade assessment of 
sex biases in placement of sociology Ph.D.s: fome evidence for 
contextual variation," ^ir^erlcan Sociolocrist, 15 (August 1980). 
120-27. 

The authors attempt to assess the effects of affirmative 
action on hiring of new faculty in sociology departments. 
Thev find that gains have been made by women, but that 
discrimination still exists in the most prestigious 
departments . 

Midom Cathv Spatz and Burke, Barbara W. , "Performance, 
aiti?idesrand professional Socialization of women in academxa," 
Sex Roles . 4 (August 1978): 549-62. 
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Investigates social psychological factors related to 
male-female productivity differences and other aspects of 
- academic life* The authors conclude that " (d) if f erences in 
the early socialization of men and women may result in the 
differential professional socialization of female faculty • 
Subtle areas of sexual discrimination in the academic 
experience are suggested" (emphasis in the original) , The 
authors do not reveal the departmental affiliations of their 
subjects. 

Wiegers, Rebecca H* and Frieze, Irene Hanson, Gender, female 
traditionality, achievement level, and cognitions of success and 
failure," Psvcholocrv of Women Quarterly , 1 (Winter 1977): 125-37. 

Attempts to ascertain the effects on the experience of 
succes or failure of gender, previous achievement level, 
and, with women, traditional or nontraditional outlooks (the 
latter being defined by college and career aspirations) • 

Zarega, Margaret E. and Walberg, Herbert J., "School science and 
femininity," in Steinkamp, Marjorie W. and Maehr, Martin , 
eds*, Women in Science (Greenwich, Conn,: JAI Press, 1984): 
37-50. Shelf, 

Studies the attitudes of young adolescent boys and girls 
toward science; proposes educational approaches to encourage 
girls * interest ' in science • 

Zuckerman, Harriet and Cole, Jonathan , "Women in American 
science," Minerva , 13 (Spring 1975): 82-102* 

An early summary of research on the obstacles which women 
face in attempting to enter the scientific profession. 
Proposes that women scientists are "triply handicapped: 
first by having to overcome barriers to their entering 
science, second by the psychic consequences of perceived 
discrimination — limited aspiration — and third by actual 
discrimination in the allocation of opportunities and 
rewards • " 
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Appendix 

The following books and papers were brought to the compiler's 
attention; after the body of the bibliography was completed. The 
section.in which the work should appear is indicated to the left 
of each set of references. 

■ ^ / 

I.A. Garfield, Eugene, "Is citation analysis a legitimate 
evaluation tool?" Scientometrics , 1 (1979): 359-375. 

Pinski, G,r "Citation based measures of research 
interactivity," Scientometrics , 2 (1980): 257-263. 

X.h. Boor, M.y "The citation impact factor: Another dubious index 
of joursxal quality," American Psychologist . 37 (1982): 975- 
977- 

Z.C* Small, Hefiry G., "Cited documents as concept symbols, " 
Social Studies of Science . 8, (1978): 327-340. 

IS. Creftwll, J.W.r Faculty Research Performance: Lessons From 
the' Sciences and the Social Sciences (Washington, D.C.: 
Association for the Study of Higher Education, 1985). 

Huhl:er » J . E . , Schmidt , F . and Jack s on , G . , Meta-Analvsis : 
Cumulating Research Findings Across Studies (Beverly Hills, 
X; Calif;i : Sage, 1982) . 

Mahpney, M.J. , "Publication prejudices: An experimental 
st:ti(ily of confirmatory bias in the peer review system," 
Coohitive Therapy and Research . 1 (1977) : 161-175. 

Martin, Ben R. , Irvine, J., Peacock, T. and Abraham, J., "A 
re-evaluation of the contributions to radio astronomy of the 
V Nancay Observatory, 4S Review (Society for the Social 

Studies of Science) I 3,- (1985) : 4-18; accompanied by comments 
by. oillmor and Gouguenheim. 

VIII.D. Boruch, R.F,, Wortman, P.M. and Cordray, D.S., eds . , 

jj ^earialvzina Program Evaluations (San Francisco: Jossey- 
Bass, 1981) . 

Cook, T.D,, Leviton, L.C. and Shadish, W.R. , "Program 
Evaluation, " J.n G. Lindzey and E. Aronson, eds.. Handbook 
of Social Psychology > 3rd ed. (N.Y.: Random House, 1985): 
699^-777. 

Cook, T.D, and Shadish, W.R. , "Metaevaluation: An 
asseiisment of ; the congressionally mandated evaluation 
^^^^Byst(im for community mental health centers," in G.J. 
St^hler and W.R. Tash, eds.. Innovative Approaches to 
Mental Health Evaluation (N.Y.: Academic Press,' 1982). 
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Cook, T.D. and Shadish, W.R., "Program evaluation: The 
worldly science," Annual Review of Psychology , 37 (1986) 
193-232. 

House, E.Rw Evaluating With Validity (Beverly Hills, 
Calif . : Sage, 1980) • 

Rossi, P.H. and Freeman, H.E., E'^valuation ; A Systematic 
Approach , 3rd. ed. (Beverly Hills, Calif.: Sage, 1985). 

Scriven, M.S., The Logic of Evaluation (Inverness, 
Calif . : Edgepress , 1980) . 

Shadish, W.R., "Sources of evaluation practice: Needs, 
purposes, questions, and technology," in L. Bickman and 
D.L. Weatherf ord, eds . , Evaluating Early Intervention 
Programs for Severely Handicapped Children and their 
Families (Austin, Texas: Pro-Ed, 1985). 

IX. MacRoberts, M.H. and MacRoberts, B.R., "Communication in 
science," Social Studies of Science , 16 (1986): 151-172. 
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